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Executive Summary 


About the Study 

Building on a previous randomized experiment of the impact of Early Colleges (ECs) (Berger et 
al., 2013), this follow-up study assessed longer-term impacts of ECs on students’ postsecondary 
outcomes 6 years after expected high school graduation. It also explored the extent to which 
students’ high school experiences mediate EC impacts. Specifically, this study addressed three 
research questions: 


1. Did EC students have better postsecondary outcomes (i.e., college enrollment and degree 
attainment) than control students? 


2. Did the impacts of ECs vary by student background characteristics (i.e., gender, 
race/ethnicity, low-income status, and prior mathematics and English language arts [ELA] 
achievement)? 


3. Were the impacts of ECs mediated by students’ high school experiences (i.e., college credit 
accrual during high school, instructional rigor, college-going culture, and student supports)? 


To answer these questions for the follow-up study, we analyzed 4 more years of postsecondary 
outcome data from the StudentTracker Service at the National Student Clearinghouse for 
students participating in the EC admission lotteries that were the basis of the previous impact 
study. We also analyzed data on student background characteristics from administrative 
records and data on high school experiences from a student survey administered in the 
previous impact study 5 or 6 years after students entered the ninth grade. 


The sample for both the original impact study and the follow-up study included 10 ECs that 

(1) enrolled students in Grades 9-12, (2) had high school graduates by 2011, (3) used lotteries 
in their admission processes for at least one of three incoming student cohorts (i.e., students 
who entered ninth grade in 2005—06, 2006-07, or 2007-08), (4) retained the lottery records, 
and (5) implemented the EC as a whole-school program. Eight of these ECs partnered with 2- 
year colleges, and two partnered with 4-year colleges. The main impact analysis sample 
included 2,458 students in total, 1,044 of whom won the EC lottery (i.e., EC students) and 1,414 
of whom did not win the EC lottery (i.e., control students). Study participants were followed 
from the year they entered Grade 9 (i.e., Year 1) to 6 years after expected high school 
graduation (i.e., Year 10). The mediating role of high school experiences was examined within 
eight of the 10 ECs that participated in the student survey administered in the previous impact 
study, with 1,424 students in total (771 EC students and 653 control students). 
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Key Study Findings 


Impact on College Enrollment 


e EC students were significantly more likely than control students to enroll in college each 
year between the fourth year of high school and 6 years after expected high school 
graduation (i.e., Year 10). Within 6 years after expected high school graduation, 84.2% of 
EC students had enrolled in college, compared with 77.0% of control students (see Exhibit 
ES.1). 


Exhibit ES.1. Percentage of Students Who Enrolled in Any Colleges Between Year 4 and Year 
10, by Study Group 
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Notes. N = 2,458 (1,044 EC, 1,414 control). The percentages for EC students are unadjusted percentages; the 
percentages for control students were computed based on the unadjusted percentages for EC students and 
estimated EC effects. 

Year x, x = 4~10, refers to x years after starting high school. 

*** 9 < 001. 


e EC students were significantly more likely than control students to enroll in 2-year 
colleges each year between the fourth year of high school and 6 years after expected high 
school graduation. By Year 10, 65.8% of EC students had enrolled in 2-year colleges, 
compared with 46.8% of control students. 


e ECstudents and control students were similarly likely to enroll in 4-year colleges and 
selective 4-year colleges over time. We did not observe a significant difference in 
enrollment in 4-year colleges after Year 6. 
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Impact on Degree Completion 


e EC students were more likely than control students to complete a postsecondary degree 
each year between the fourth year of high school and 6 years after expected high school 
graduation. By Year 10, 45.4% of EC student and 33.5% of control students had completed a 
certificate, associate’s degree, or a bachelor’s degree (see Exhibit ES.2). 


Exhibit ES.2. Percentage of Students Who Completed Any Postsecondary Degree Between 
Year 4 and Year 10, by Study Group 
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e EC students were more likely than control students to complete an associate’s degree or 
certificate each year between the fourth year of high school and six years after expected 
high school graduation. By Year 10, 29.3% of EC student and 11.1% of control students had 
completed an associate’s degree or certificate. 


e ECstudents were more likely than control students to complete a bachelor’s degree each 
year between the second and sixth years after expected high school graduation. By Year 
10, 30.1% of EC student and 24.9% of control students had completed a bachelor’s degree. 


Differential EC Impact 


e EC impacts on college enrollment and degree completion outcomes were similar for 
students with different family background characteristics. EC impacts on college 
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enrollment and degree completion outcomes did not differ significantly by gender, 
race/ethnicity, or eligibility for free or reduced-price lunch. 


e ECimpacts on some postsecondary outcomes were stronger for students with higher 
levels of Grade 8 achievement. EC impacts on enrollment in 2-year colleges and completion 
of an associate’s degree or certificate within 6 years after expected high school graduation 
(i.e., by Year 10) were stronger for students with higher levels of Grade 8 achievement. EC 
impacts on enrollment in 4-year colleges and bachelor’s degree completion did not 
significantly differ by Grade 8 achievement. 


The Mediating Role of High School Experiences 


e High school experiences significantly explained the EC impact on enrollment in any type of 
institution. High school experiences, as measured by instructional rigor, college-going 
culture, and student supports, explained approximately 30% the EC impact on enrollment in 
any type of institution within 6 years after expected high school graduation (i.e., by Year 
10). These high school experience measures did not explain the EC impact on enrollment in 
2-year colleges. 


e College credit accrual during high school was the strongest mediator for degree 
completion outcomes, particularly bachelor’s degree completion. Completion of college 
credits during high school explained approximately 87% of the EC impact on bachelor’s 
degree completion within 6 years after expected high school graduation (i.e., by Year 10). 


Summing Up 

Adding to the positive findings about the EC impacts on students’ high school graduation and 
postsecondary outcomes from the original impact study, this follow-up study generated strong 
evidence for the longer-term impacts of ECs on students’ postsecondary outcomes. We found 
that EC students were more likely to enroll in and graduate from 2-year colleges, which may 
partly be because most of the ECs in our study were partnered with 2-year colleges. While EC 
students were no more likely than their peers to enroll in 4-year colleges, they were more likely 
to complete a bachelor’s degree and did so earlier in their academic careers. These positive 
impacts were similarly observed for students from different family backgrounds. 


Because EC students were more likely to complete college degrees earlier than their peers, we 
anticipate positive long-term EC impacts on students’ workforce and financial outcomes. 
While more research is needed to explore the EC impacts on later-life outcomes, the 
accelerated timeline of degree attainment for EC students combined with the fact that college 
credits earned at ECs come at little or no cost to them suggest that EC students may accrue less 
educational debt in their lives. Moreover, as EC students are likely to enter the workforce 
sooner, we expect that they will have higher lifetime earnings compared with their peers. 
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Where the mechanisms of EC impacts are concerned, we found that students’ high school 
experiences explained EC impact on overall college enrollment, but they did not explain the EC 
impact on enrollment in 2-year colleges. This is likely because most EC students enrolled in 
partnering 2-year colleges while in high school, and their high school experiences were more 
likely to affect their later outcomes than their outcomes during high school. However, 
accumulation of college credits during high school explained most of the EC impact on 
bachelor’s degree completion, even though the credits were earned at a 2-year college for the 
majority of students. This finding speaks to the important role that 2-year colleges can play in 
the academic trajectories of high school students, when the integration of college coursework 
into the high school curriculum is intentional and coheres to specific degree-attainment 
pathways. 
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Introduction 


There is substantial evidence that a postsecondary degree or credential prepares students for 
successful entry into the workforce. Bachelor’s degree holders earn more over a lifetime than 
individuals with only a high school diploma (Carnevale, Rose, & Cheah, 2011), and college 
degree earners fared better in the recent American recession than adults with only a high 
school diploma (Grusky, Bird, Rodriguez, & Wimer, 2013). Moreover, workforce projections 
consistently predict that the lion’s share of future jobs will require a postsecondary degree 
(Carnevale et al., 2011). 


During the past decade, a growing body of research evidence has emerged noting the promise 
of dual enrollment as an effective way to promote postsecondary access and success. While the 
implementation of dual enrollment programs varies across schools and may be dictated by 
state policies, dual enrollment is generally defined as students’ participation in college-level 
courses that count for credit at both the secondary and postsecondary levels. A recent evidence 
review conducted by the What Works Clearinghouse (WWC, 2017a) concluded that dual 
enrollment programs had positive effects on high school achievement, high school graduation, 
credit accumulation, college enrollment, and degree completion outcomes. While the concept 
of dual enrollment has been around for several decades, it is perhaps due to the evidence of 
the effectiveness of these programs that there continues to be a push at both the state and 
federal levels to support partnerships between high schools and postsecondary institutions to 
provide students with the opportunity to earn college credits during high school (Achieve & 
Jobs for the Future, 2015; Zinth, 2016). 


One type of dual enrollment program that has received much attention and has been 
expanding rapidly across the nation is Early Colleges (ECs). ECs were originally created as part of 
the Early College High School Initiative (ECHSI) established in 2002 by the Bill & Melinda Gates 
Foundation, along with the Carnegie Corporation of New York, the Ford Foundation, and the 
W.K. Kellogg Foundation. The explicit goal of the initiative was to increase the opportunity for 
students who are disadvantaged to earn a postsecondary credential. To achieve this goal, ECs 
partner with colleges and universities to offer students an opportunity to earn an associate’s 
degree or up to 2 years of college credits toward a bachelor’s degree during high school at no 
or low cost to their families. ECs also provide a rigorous and supportive high school 
environment to help students navigate and succeed in college coursework. More specifically, 
the ECHSI was guided by the following five core principles: 


1 ECs employ a variety of approaches to cover students’ tuition costs, including tuition payments from the state or 
district or a college’s decision to waive tuition for EC students. Funding availability and local/state policy contexts, 
however, require that students in some states absorb some or all of this expense (Berger et al., 2009). 
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1. ECs are committed to serving students underrepresented in higher education. 


2. ECs are created and sustained by a local education agency, a higher education institution, 
and the community, all of whom are jointly accountable for student success. 


3. ECs and their higher education partners and community jointly develop an integrated 
academic program so all students earn one to two years of transferable college credit 
leading to college completion. 


4. ECs engage all students in a comprehensive support system that develops academic and 
social skills as well as the behaviors and conditions necessary for college completion. 


5. ECs and their higher education and community partners work with intermediaries to create 
conditions and advocate for supportive policies that advance the Early College movement 
(Jobs for the Future, 2008, p. 2). 


The underlying assumption of the ECHSI is that engaging students who are underrepresented in 
a rigorous high school curriculum tied to the incentive of earning college credit (with reduced 
financial burden) will increase their access to and success in postsecondary education. Exhibit 1 
illustrates the ECHSI theory of change. 


Exhibit 1. ECHSI Theory of Change 
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Between 2010 and 2013, the American Institutes for Research (AIR) conducted an impact 
evaluation of ECs based on a natural experiment, capitalizing on retrospective admission 
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lotteries that occurred between 2005-06 and 2007-08 (Berger et al., 2013; Berger, Turk- 
Bicakci, Garet, Knudson, & Hoshen, 2014; Haxton et al., 2016). Among the lottery participants, 
those who won the lottery and were offered admission were considered treatment students; 
those who did not win the lottery were considered control students. By means of lottery-based 
random assignment, any observed outcome differences between the treatment and control 
students could be attributed exclusively to whether they were offered admission to an EC. The 
study found that being offered admission to an EC had positive impacts on both college 
enrollment and degree attainment 2—4 years after expected high school graduation. 


This report presents the findings from a follow-up study designed to assess the longer-term 
impacts of ECs by extending the previous AIR study with 4 more years of student outcome data. 
Focusing on the impacts of ECs on students’ postsecondary outcomes 6 years after expected 
high school graduation, the follow-up study addressed the following research questions (RQs): 


e RQ1: Did EC students have better postsecondary outcomes (i.e., college enrollment and 
degree attainment) than control students? 


e RQ2: Did the impacts of ECs vary by student background characteristics (i.e., gender, 
race/ethnicity, low-income status, and prior mathematics and English language arts [ELA] 
achievement)? 


In addition to the overall impact (RQ1) and potential differential impact (RQ2) of ECs, this 
follow-up study also explored the role of students’ high school experiences as potential 
mediators of EC impact, which had not been examined in prior research and which may provide 
useful insights into the “black box” of ECs. The mediation analyses addressed the following RQ: 


e RQ3: Were the impacts of ECs mediated by students’ high school experiences (i.e., college 
credit accrual during high school, instructional rigor, college-going culture, and student 
supports)? 


Before presenting detailed findings for each RQ, we provide a brief review of relevant literature 
and a description of the methods that we used to address the RQs. 


Literature Review 


A variety of instructional practices and programs aim to improve students’ college and career 
readiness through “accelerated learning,” or programs that offer high school students exposure 
to college-level instruction (Hodara & Pierson, 2018). One of the most prevalent programs for 
accelerating students’ learning is dual enrollment, which involves students’ participation in 
college-level coursework during high school, allowing students to meet high school and college 
course requirements simultaneously. ECs are a type of dual enrollment program where 
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(1) all students within the high school are expected to take college-level courses and (2) 
students follow a prescribed curriculum that facilitates the accrual of 2 years of college credit, 
and in some cases the completion of an associate’s degree, by the time students graduate from 
high school. 


Qualitative studies of dual enrollment and ECs have found that these opportunities influence 
the creation of an educational identity and provide supportive and caring relationships 
between students and teachers, particularly for students who are traditionally underserved in 
higher education (Beall, 2016; Kanuika & Vickers, 2010; Pitchford-Nicholas, 2015; Thompson & 
Onganga, 2011; Wolk, 2005). ECs may improve the “clarity of the college-student role,” by 
providing students with a more concrete vision of who is able to succeed in college and which 
skills are required, allowing students to identify themselves as college ready (Lile, Ottusch, 
Jones, & Richards, 2018; Newton, 2008; Newton & Vogt, 2008). Rosenbaum and Becker (2011) 
similarly highlighted the importance of motivating students by creating educational identities 
and providing frequent opportunities to identify academic strengths and receive the supports 
they need to succeed. Therefore, as illustrated in the ECHSI theory of change, dual enrollment 
and EC programs have the potential to positively influence student outcomes by providing 
students with opportunities to interact with college staff and other college students, exposing 
them to a rigorous curriculum and an environment where all students are expected to attend 
college, and making sure they receive the support they need to succeed. 


Below, we provide a summary of the research on the impact of dual enrollment programs and 
ECs on students’ secondary and postsecondary outcomes. 


Research on Dual Enrollment 

Positive effects of dual enrollment programs on student outcomes have been observed in a 
range of research studies, from correlational research (Karp, Calcagno, Hughes, Dong, & Bailey, 
2007; Swanson, 2008) to quasi-experimental studies (An, 2013a; Blankenberger, Lichtenberger, 
& Witt, 2017; Giani, Alexander, & Reyes, 2014; Miller et al., 2018; Speroni, 2011; Struhl & 
Vargas, 2012). While some of these studies focused on specific states (e.g., Texas, North 
Carolina, Indiana) in which education policy requires or facilitates partnerships between high 
schools and postsecondary institutions (Edmunds et al., 2017; Indiana Commission for Higher 
Education, 2016; Miller et al., 2018; Struhl & Vargas, 2012), other studies found similarly 
positive effects of dual enrollment using nationally representative samples of students (An, 
2013a; Swanson, 2008). Overall, evidence of the academic benefits associated with dual 
enrollment has been consistent across study designs, study sites, and types of dual-enrollment 
programs. 
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Two seminal correlational studies show that students who took college classes during high 
school had higher high school grade point averages; were more likely to receive high school 
diplomas, enroll in college, enroll in college full time, and persist in college; and earned more 
college credits after high school graduation than students without college experience during 
high school (Karp et al., 2007; Swanson, 2008). Building off this work, more recent research 
using quasi-experimental methods found that dual enrollment participation increased the 
probabilities of college attendance and degree completion (An, 2013a; Struhl & Vargas, 2012). 


Research also demonstrates positive relationships between dual enrollment and postsecondary 
outcomes for students who are traditionally underrepresented in college. For example, dual 
enrollment was found to be positively associated with college enrollment, grade point average, 
and degree completion for low-income students and lower-achieving students (An, 2013b; 
Community College Research Center, 2012). In addition, a study on the Concurrent Course 
Initiative in California, a career-focused dual enrollment program targeting students 
traditionally underrepresented in college, found that program participants had higher rates of 
high school graduation, college enrollment, and college persistence; accumulated more college 
credits; and required less remediation than nonparticipants (Hughes, Rodriguez, Edwards, & 
Belfield, 2012; Rodriguez, Hughes, & Belfield, 2012). While some studies found that the effects 
of dual enrollment are similar for students from different socioeconomic backgrounds (e.g., An, 
2013a), one study conducted in Illinois found that the positive effects of dual enrollment on 
postsecondary enrollment and degree completion outcomes were smaller for minority students 
and low socioeconomic status students than the average effects observed within the state 
(Taylor, 2015), suggesting that dual credit programs may need to provide extra support to 
students who are traditionally underserved in higher education in order to provide similar 
benefits. 


In addition, some research found mixed effects of dual enrollment programs for lower- 
achieving students: while lower-achieving students participating in dual enrollment programs 
may be more likely to enroll in college after high school than their peers that do not participate 
in dual enrollment (Cowan & Goldhaber, 2015; Karp et al., 2007), the rigor of dual enrollment 
may also be associated with increased incidence of dropout for lower-achieving students 
(Cowan & Goldhaber, 2015). These findings suggest that the implementation of dual 
enrollment in the absence of the schoolwide focus on college readiness and individually 
tailored supports for struggling students, such as those exhibited by ECs, may have unintended 
negative consequences for traditionally underserved students. 


Research on ECs 


Since 2002, more than 280 ECs have opened nationwide as part of the ECHSI, serving more than 
80,000 students in 31 states and the District of Columbia (Webb, 2014). The number of ECs 
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outside of the ECHSI also has continued to increase across the country, as a growing body of 
research evidence has emerged noting the promise of ECs as an effective way to promote 
postsecondary access and success. The most comprehensive study of ECs to date is a 6-year 
national evaluation of ECHSI conducted by Berger et al. (2009), which collected data on all 

157 ECs open nationwide as of fall 2007. The evaluation concluded that EC students performed 
better than students in other high schools located in the same district. It found, for example, 
that EC students outperformed their peers in the same district by 7 percentage points on 
average in terms of proficiency rate on state assessments in both English language arts and 
mathematics. It also estimated (based on progression rates) that 66% of the students who 
started at an EC in ninth grade would graduate on time, which was 14 percentage points higher 
than the estimated graduation rate for other high schools in the local districts. 


Prior research also shows that EC students performed favorably compared with national figures. 
Drawing on data from a representative sample of 100 ECs, a 2014 report released by Jobs for the 
Future, for example, reveals that EC students were “far more likely to graduate high school” — 
90% of EC students received a high school diploma vs. 78% of students nationally—despite the 
fact that ECs serve primarily low-income students and students of color (Webb, 2014, p. 10). 
The report also shows that EC students were more likely to earn college credits in high school 
(94% of EC students vs. less than 10% nationally), more likely to complete a college degree by 
high school graduation (30% of EC students vs. very few students nationally), more likely to 
enroll in college immediately after high school (71% of EC graduates vs. 54% of low-income high 
school graduates nationally), and more likely to return to college for a second year (86% of EC 
graduates who enrolled in college vs. 72% of college students nationally). 


Most of the existing studies of ECs, however, are descriptive or qualitative in nature and do not 
warrant causal conclusions about the impact of ECs. One exception is a study that used 
propensity score matching to assess the EC impact on students’ college preparedness in 
mathematics and science by comparing students attending 33 ECs in North Carolina with similar 
students attending other high schools in the same districts (Miller & Corritore, 2013). The study 
found that attending an EC had positive effects on both students’ progression through the 
mathematics pipeline (which consisted of three college-preparatory courses) and their 
mathematics performance, but nil to negative effects on students’ science pipeline progression 
and no effect on their science performance. Also relying on propensity score matching, a more 
recent study examined the performance of students at all 70 of the ECs in North Carolina 
relative to a matched sample of students who attended the same middle school but did not 
attend the EC during high school (Lauen, Fuller, Barrett, & Janda, 2017). This study found 
significantly better outcomes for EC students in both high school and college, including lower 
ninth-grade retention rates, higher test scores, fewer absences, higher graduation rates, higher 
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rates of enrollment at 4-year state colleges, and higher rates of completing an associate’s 
degree within 2 or 3 years of high school completion. 


To date, the most rigorous evidence on the impact of ECs came from two natural experiments 
based on admission lotteries. One is the AIR study mentioned earlier (Berger et al., 2013, 2014; 
Haxton et al., 2016) that is the basis of the follow-up study presented in this report. The other is 
a study conducted by the SERVE Center at the University of North Carolina at Greensboro 
(Edmunds et al., 2012, 2017; Edmunds, Willse, Arshavsky, & Dallas, 2013). Both studies found 
positive impacts of ECs on a variety of student outcomes both during and after high school. 


Similar to the AIR study that is based on admission lotteries taking place in 10 sites, the SERVE 
Center study is based on admissions lotteries taking place in 12 sites. It found that EC students 
were more likely to be “on track for college” than control students, and that ninth-grade EC 
students were more likely than control students to take core college preparatory courses and 
succeed (Edmunds et al., 2012). The study also found that EC students had higher attendance 
rates, lower suspension rates, and higher levels of engagement than control students (Edmunds 
et al., 2013). The most recent report from the SERVE study further shows that EC students 
accrued significantly more college credits while in high school (21.6 vs. 2.8 credits), and 
graduated from high school (85.4% vs. 81.4%), enrolled in postsecondary institutions (89.9% vs. 
74.3%), and received postsecondary credentials (30.1% vs. 4.2%) at higher rates than control 
students within 6 years of starting high school (Edmunds et al., 2017). 


Overall, the available research evidence on ECs accumulated over the past decade is quite 
encouraging. Nevertheless, the evidence base for the EC impact on student outcomes is still 
understandably thin, given the relatively short history of ECs. Moreover, none of the prior studies 
have examined the EC impact on longer-term student outcomes beyond 2 years after expected 
high school graduation. By extending the previous AIR EC impact study with 4 more years of 
postsecondary data, the follow-up study presented in this report represents a significant addition 
to the existing evidence base on the impact of ECs on longer-term student outcomes. 


Methods 


Sample 

The original impact study of the ECHSI (Berger et al., 2013) that is the basis for this follow-up 
study was a multisite natural experiment with student-level random assignment based on 
retrospective lotteries that occurred prior to the start of the study. The study sought ECs that 
conducted admission lotteries long enough in the past so that students who participated in the 
lotteries would have had the opportunity to graduate from high school and enter 
postsecondary education by the time data collection concluded. The retrospective feature of 
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the original study made it possible to estimate the EC impact on postsecondary outcomes 
within a relatively short time frame. 


Specifically, to be eligible for the retrospective study, an EC had to meet the following criteria: 
(1) enrolled students in Grades 9-12, (2) had high school graduates by 2011, (3) used lotteries 
in its admission processes for at least one of three incoming student cohorts (i.e., students who 
entered ninth grade in 2005-06, 2006-07, or 2007-08), (4) retained the lottery records, and (5) 
implemented the ECHSI as a whole-school program. The study sample was restricted to ECs that 
were open by fall 2007 to ensure that students in the study would have had the opportunity to 
complete at least 2 years of college after expected high school graduation by the end of the 
original study (2013). For this follow-up study, we collected additional years of postsecondary 
data so that we would be able to examine student outcomes for 6 years following expected 
high school graduation for all three student cohorts included in the original impact study. 
Exhibit 2 illustrates the expected educational progression during the years covered by the 
original study and the follow-up study for each of the three student cohorts. 


Exhibit 2. Students’ Expected Educational Progression During the Periods Covered by the 


Original Impact Study and the Follow-up Study, by Cohort 


Period Covered by the Original Impact Study Period Covered by the Follow-up Study 


Grade 9 4 years post HS 6 years post HS 
2005-06 (Y1) 2012-13 (Y8) 2014-15 (Y10) 
Cohort 2 


Grade 9 3 years post HS 6 years post HS 
2006-07 (Y1) 2012-13 (Y7) 2015-16 (Y10) 


Grade 9 2 years post HS 6 years post HS 
2007-08 (Y1) 2012-13 (Y6) 2016-17 (Y10) 


Notes. HS = high school. Yx, x = 1-10: x years since starting high school. This exhibit depicts students’ expected 
educational progression, assuming a 4-year high school program. Students may take more or less time to complete 
high school (two of the ECs in the study sample had 5-year programs). 


Of the 154 ECs open nationwide by fall 2007, 10 met the criteria for inclusion in the original 
impact study. (See Appendix A for a CONSORT diagram illustrating sample selection.) These 10 
schools are located in five states (i.e., North Carolina, Ohio, South Carolina, Texas, and Utah); 
five schools are located in urban areas, two in midsized cities, and three in small towns. Nine 
schools opened as new schools, and one was an existing school that became an EC. Eight of the 
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ECs had partnerships with 2-year colleges, and the other two had partnerships with 4-year 
colleges. All 10 schools were small schools (i.e., fewer than 150 students per grade) with 
enrollment ranging from about 100 to 600 students, with an average enrollment of 290. Across 
the 10 schools, 49% of the students were non-White (ranging from 12% to 100%), and 44% 
were eligible for free or reduced-price lunch (ranging from 9% to 99%) based on 2007-08 data. 


Of the 10 ECs identified for the original study, 6 conducted admission lotteries for more than one 
student cohort, and 3 conducted multiple lotteries (i.e., “sublotteries”) in a given year, such as 
separate lotteries for applicants from different feeder schools or districts. The original study 
included all students who participated in the 23 lotteries (including sublotteries) across three 
cohorts conducted by the 10 ECs, with students who were offered admission to the ECs identified 
as treatment students and students not offered admission identified as control students. In total, 
the study included 2,458 students (1,044 EC/treatment students and 1,414 control students). (See 
Appendix B for the number of students by condition in each lottery.) The control students were 
spread across 272 different high schools during the 4 years after participating in the EC admission 
lotteries, with many of those schools enrolling only one or two control students. 


The background characteristics of the students who participated in our study are presented in 
Exhibit 3. Approximately half of the sample was female, half was non-White, and half was low 
income (i.e., eligible for free or reduced-price lunch). Fewer than a quarter of study participants 
had parents who did not attend college (i.e., first-generation college goers). Students who 
participated in the EC lotteries had average Grade 8 ELA and mathematics test scores that were 
above the state average. A comparison of the background characteristics between EC students 
and control students revealed small differences between groups, none of which are statistically 
significant at the .05 level, indicating that the two groups were equivalent at baseline. 


Exhibit 3. Background Characteristics of EC Students and Control Students in the Impact Study 
Sample 


EC Students 
(N= 1,044) 


Control Students 
(N =1,414) 


Group Difference 
(Effect Size) 


Student Characteristic 


Female 


Nonwhite 


Low-income 


First-generation college-going 


Grade 8 ELA test score 


Grade 8 mathematics test score 


Notes. The means for EC students are unadjusted means; the means for control students were computed by subtracting 
the estimated group mean difference from the unadjusted means for EC students. All baseline equivalence tests were 
conducted using two-level models that were similar to the main impact model (see Appendix D). 
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Data Sources 

Data for the original study came from a variety of sources. Information about who participated in 
the EC admission lotteries and who was offered admission to the EC was obtained from the ECs. 
Students’ demographic information and achievement on Grade 8 state assessments were obtained 
from participating districts and states. Data for students who applied to ECs in North Carolina came 
from a longitudinal experimental study on ECs led by the University of North Carolina at 
Greensboro SERVE Center. The student demographic and prior achievement data collected in the 
previous study continue to be available for this follow-up study for all but one EC, which is in a state 
that no longer allows researchers access to identifiable student-level demographic and 
achievement data. Demographic and prior achievement data for all EC and control students from 
this site were imputed for the follow-up study.” While we included this site in the main impact 
analyses, we performed a set of sensitivity analyses in which this site was excluded.? 


In addition to extant administrative records, the original study collected information about 
students’ perceptions of their high school experiences through a student survey conducted in 
winter 2011/spring 2012. The survey was given only to the two youngest cohorts (i.e., students 
who entered ninth grade in 2006-07 and 2007-08) because pilot work indicated that older 
students were harder to locate and had difficulty remembering details of their high school 
experiences. Eight of the 10 ECs in the study sample participated in the survey; the other two 
ECs did not participate because they could not provide the student contact information needed 
to administer the survey. Based on a power analysis conducted during the design phase of the 
original study, 1,424 students (771 EC students and 653 control students) from the eight ECs 
were sampled for the survey, representing 58% of the total study sample. The survey response 
rate was 93% for the EC students and 87% for the control students, with an overall response 
rate of 90%. (See Appendix B for the number of survey respondents for each lottery.) 


Finally, the original study obtained data on students’ enrollment in postsecondary education and 
degree completion as of fall 2013 using the StudentTracker Service at the National Student 
Clearinghouse (NSC). This follow-up study collected additional years of NSC data in winter 2017, 
allowing for between 6 years (for Cohort 3 students) and 8 years (for Cohort 1 students) of data after 
expected high school graduation for students in our study sample. The NSC collects data on student 
enrollment and degree completion from more than 3,600 degree-granting higher education 
institutions across the United States, and it currently covers more than 98% of all student 
enrollments in public and private colleges and universities in the United States (NSC, 2017). For a 


* Most of the students from this site took the student survey, and the available survey data were used to impute 
student demographic and prior achievement data for this site. 

3 In addition, for outcome measures that are common between the original EC impact study (Berger et al., 2013) and this 
follow-up study, we compared results between the original study (for which we had unimputed data for this site) and 
the follow-up study (for which we had imputed data for this site). Overall, results were similar between studies. 
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small percentage of college students, the NSC cannot provide their postsecondary data because their 
colleges did not report data to the NSC’ or because the students refused to allow their records to be 
released (Dynarski, Hemelt, & Hyman, 2015). Because the NSC data do not allow researchers to 
distinguish a student who did not attend college from a student with missing records for the reasons 
stated, estimates of college enrollment rates and degree completion rates based on the NSC data 
may be slightly lower than the actual rates. However, given that the number of missing records from 
NSC is very small, their influence on the results from this study is likely to be negligible. 


Measures 
In this section, we provide a description of all of the measures used in our statistical models. 


Outcome Measures 
This follow-up study examined student outcomes in two outcome domains with three primary 
outcomes in each domain: 


1. Outcomes in the College Enrollment Domain: Enrolled in college, enrolled in a 2-year college, 
and enrolled in a 4-year college within 6 years after expected high school graduation (i.e., by 
Year 10). 


2. Outcomes in the Degree Attainment Domain: Completed any postsecondary degree, 
completed an associate’s degree or a certificate, and completed a bachelor’s degree within 
6 years after expected high school graduation (i.e., by Year 10). 


For the three primary outcomes in the degree attainment domain, we used “within 6 years after 
expected high school graduation” as the time frame, as it is an approximation of the standard 
time frame (i.e., 150% of normal time of degree completion) that colleges use to report to the 
National Center for Education Statistics (NCES) on the completion of bachelor’s degrees in 
compliance with the 1990 Student Right to Know Act.° In the NCES Integrated Postsecondary 
Education Data System (IPEDS), the time frame for degree completion begins when students 
enroll in college for the first time rather than when they graduate from high school. Therefore, 
our measure of degree completion is equivalent to the IPEDS measure of degree completion for 


4 In fall 2009, which is the fall immediately after expected high school graduation for the oldest student cohort in 
our study, the NSC obtained enrollment data from 96.5% of all public 4-year colleges and 94.6% of all public 2-year 
colleges nationally. The coverage was lower in the private sector: 85.3% of private 4-year colleges and 14.8% of 
private 2-year colleges. Even though the NSC coverage of private 2-year colleges is low, it should have only minimal 
influence on estimated enrollment or degree completion rates because enrollment in this sector represents only 
approximately 2% of the total postsecondary enrollment in the nation. 

5 NCES also reports rates of degree completion both within 100% of normal time and within 200% of normal time 
of degree completion. 
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students who graduated from high school within 4 years of high school entry and entered college 
for the first time immediately after high school graduation.°® 


To obtain a more comprehensive picture of the EC impact on college enrollment and degree 
attainment over time, we also examined whether students enrolled in college, enrolled in a 2-year 
college, enrolled in a 4-year college, completed any type of postsecondary degree, or completed an 
associate’s degree or a certificate by the end of each year from Year 4 after entering ninth grade 
through 6 years after expected high school graduation (i.e., Year 10).’ Because students are not 
expected to earn more than 2 years of college credit during high school, we examined whether 
students completed a bachelor’s degree between 2 and 6 years after expected high school 
graduation. (See Appendix C for a list of all outcome measures examined in this study with their 
definitions.) 


The measures of college enrollment described above are not mutually exclusive; over the period 
covered by this study, students may enroll in both 2-year and 4-year colleges. To provide a detailed 
snapshot of students’ postsecondary enrollment status within each year between Year 5 (i.e., the 
first year after expected high school graduation) and Year 10 (i.e., the sixth year after expected high 
school graduation),® we also examined a series of categorical measures of students’ enrollment 
profiles at each point in time. Within each given year, students were classified into one of four 
mutually exclusive categories: (1) not yet enrolled in college, (2) enrolled exclusively in 2-year 
colleges, (3) ever enrolled in a 4-year college with a prior enrollment in a 2-year college,’ or (4) ever 
enrolled in a 4-year college without prior enrollment in a 2-year college. Once students enrolled in a 
4-year college (either with or without a prior enrollment at a 2-year college), they remained in that 
category in subsequent years. Similarly, students who enrolled exclusively in 2-year colleges 
remained in that category in subsequent years unless they transferred to a 4-year college. 


Finally, to further explore EC impact on the types of colleges in which students enroll, we 
examined differences between EC and control students in enrollment in selective 4-year colleges 
each year from Year 4 after entering ninth grade through 6 years after expected high school 
graduation. The selectivity of postsecondary institutions was determined using information from 


5 Our measures of degree completion also differ from the IPEDS measures in that the IPEDS measures capture only 
those degrees that were obtained from the first college that students attended, whereas our measures are based 
on the NSC data that capture degrees obtained from any college that students attended. 

7 Results for completion of an associate’s degree or a certificate within 3 years after expected high school 
graduation (i.e., by Year 7) will approximate the NCES definition of associate’s degree completion (i.e., 
approximately 150% of normal time for completing an associate’s degree). 

8 For this analysis, we do not look at enrollment profiles during the fourth year of high school due to the relatively 
small number of control students who enrolled in college during high school. 

° Category (3) is intended to capture students who started at a 2-year college and then transferred to a 4-year 
college. There are a small number of students in our sample who started at a 4-year college, then went to a 2-year 
college, and later transferred back to a 4-year college. Those students were considered as belonging to Category 
(4) for the purpose of this study. 
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the Barron’s Profiles of American Colleges (2008, 2010, 2012, 2014, 2016, 2017, 2018). Barron’s 
identifies selective colleges based on the percentage of applicants who are admitted into the 
institution as well as the achievement characteristics of incoming freshmen students. For this 
study, we define selective colleges as colleges that were classified as “most competitive” or 
“highly competitive” by Barron’s in the year in which students enrolled in the college. These are 
colleges in which no more than 50% of the applicants are offered enrollment, with incoming 
students having grade point averages of a B or higher and coming from the top 35% of their high 
school class. Median SAT scores at these colleges exceed 620, and median ACT scores exceed 27. 
Each year, between 176 and 196 colleges were identified by Barron’s as “most competitive” or 
“highly competitive.” Annual information from Barron’s was merged with student data to 
determine that colleges were classified as selective during the academic year in which study 
participants were enrolled in the college. Because Barron’s Profiles of American Colleges was not 
printed every year during the period examined in this study, we were unable to obtain lists from 
Barron’s for the 2008-09, 2010-11, 2012-13, and 2015-16 academic years. For these years, we 
relied on the list of colleges from the previous year. 


Given that the study includes multiple outcomes in each outcome domain, there may be 
increased risk for inflated Type | error due to multiple comparisons. To guard against such risk, 
we have designated a single outcome in each outcome domain (i.e., ever enrolled in college 
and ever completed a postsecondary degree or certificate) as the “confirmatory” outcome, and 
our main conclusion about the treatment effect in each domain is based on the confirmatory 
outcome. To check the robustness of our findings to potentially increased Type | error due to 
multiple comparisons, we also used the Benjamini-Hochberg method to correct for multiple 
comparisons within each outcome domain, as recommended by the WWC (2017b). Because 
there are three primary outcome measures within each outcome domain, we corrected for 
three comparisons within each domain. 


Moderators and Covariates 

Given the focus of the ECHSI on students who are disadvantaged, we expect that the EC impact 
may vary across different types of students (RQ2). To test this hypothesis, we conducted 
moderator analyses with the following measures of student background characteristics as 
potential moderators: gender, race/ethnicity (White versus non-White), low-income status 
(eligibility for free or reduced-price lunch), and eighth-grade mathematics and ELA achievement 
scores (Standardized based on state means and standard deviations). Data on all these 
measures were collected from administrative records for the original impact evaluation. These 
measures also were used as covariates to improve the precision of estimates in all our analyses. 
In addition, the student survey administered for the original impact study in winter 2011/spring 
2012 asked students about their parents’ level of education. Impact models included a 
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covariate indicating whether students reported that neither of their parents attended college. *° 
Finally, while we also collected student-level data about English language learner (ELL) status 
and Individualized Education Program (IEP) status before entering high school, only a small 
percentage of students in our sample were ELLs (less than 1%) or had IEPs (7%). Therefore, we 
did not include these two variables as covariates in the impact analyses. 


Mediators 

As indicated by the EC theory of change (see Exhibit 1), we hypothesize that the EC impact on 
student outcomes is mediated by students’ high school experiences (RQ3). To test this 
hypothesis, we conducted mediation analyses with four mediators measuring key aspects of 
students’ high school experiences based on the student survey administered in winter 
2011/spring 2012 in the original impact study (see Exhibit 4). 


Exhibit 4. Measures of Students’ High School Experiences as Potential Mediators of EC Impact 


College credit accrual | A dichotomous indicator of whether a student (1) earned any college credit in high 
during high school? school or (2) passed at least one advanced placement exam 


Potential Mediator 


Instructional rigor An eight-item scale ranging from 0 (never) to 4 (almost every day) measuring the 
frequency with which students reported defending their ideas, writing five or more 
pages on a topic, explaining thinking, applying what they had learned, engaging in 
discussions, doing research, presenting work, and receiving feedback in courses that 
they took during Grades 9-12; reliability (a) = .82 


College-going culture | Athree-item scale ranging from 1 (strongly disagree) to 4 (strongly agree) measuring the 
extent to which the principal, teachers, and students expected students in the high 
school to go to college; reliability (a) = .79 


A five-item scale ranging from 1 (strongly disagree) to 4 (strongly agree) measuring the 
extent to which teachers cared if students came to school, praised students for their 
effort, listened to students, encouraged students, and cared about students; reliability 
(a) = .88 


Student support 


* Because college credit accrual during high school implies college enrollment, we examined college credit accrual 
during high school as a potential mediator only for EC impact on degree attainment outcomes, not for college 
enrollment outcomes. 


Analytic Methods 
In this section, we briefly describe the methods we used to address the three RQs guiding this 
study. Further technical details on our analytic methods can be found in Appendix D. 


10 Because of the relatively large amount of missing data for first-generation college-going status due to both 
survey nonresponse and sampling procedures (i.e., students in the oldest cohort did not participate in the student 
survey), we did not examine differences in EC impact by first-generation status. Overall, 33.3% of treatment 
students and 60.8% of control students in our study had missing data on first-generation status. 
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Main Impact Analyses 

Our main impact analyses are intent-to-treat (ITT) analyses, which estimate the impact of being 
offered admission to an EC through a lottery, regardless of whether students actually enrolled 
in the EC. To estimate overall ITT effects across lotteries, we constructed a two-level logistic 
regression model that takes into account the clustering of students within lotteries. Given the 
limited number of lotteries in the study sample, we treated lotteries as fixed effects in our main 
ITT analyses, and we also estimated random-effects models for the six primary outcomes as 
sensitivity analyses. All ITT analyses controlled for students’ demographic characteristics and 
Grade 8 achievement to improve the precision of impact estimates. 


For the categorical outcome of postsecondary enrollment profile in a given year, our ITT 
analysis was based on a multinomial logistic regression model where the outcome has four 
possible categories: (1) had not enrolled in college (the baseline category), (2) enrolled 
exclusively in 2-year colleges, (3) ever enrolled in a 4-year college with a prior enrollment in a 2- 
year college, and (4) ever enrolled in a 4-year college without a prior enrollment in a 2-year 
college as of the given year. The model estimates the EC impact on the probability of being in a 
particular outcome category relative to the probability of being in the baseline category (i.e., 
impact on the relative risk ratio), while controlling for lottery/sublottery fixed effects and 
student background characteristics. ** 


Given that not all lottery participants complied with their treatment assignment, we 
supplemented the ITT analyses with complier average treatment effect (CATE) analyses for the 
six primary outcomes measured 6 years after expected high school graduation: enrollment in 
college, enrollment in a 2-year college, enrollment in a 4-year college, completion of a 
postsecondary degree, completion of an associate’s degree or certificate, and completion of a 
bachelor’s degree. Relying on an instrumental variable approach, the CATE analyses explicitly 
take into account noncompliance with treatment assignment (i.e., no-shows and crossovers), 
and estimate the average treatment effects of actually attending an EC for students who 
complied with their treatment assignment rather than the average treatment effects of being 
offered admission to an EC through a lottery (i.e., the ITT effects). (See Appendix D for further 
details about both the ITT and CATE analyses.) 


Differential Impact Analyses 

To answer the second research question about potential differential impacts of ECs on students 
with different background characteristics, we added an interaction between treatment status 
and a given student characteristic into the student-level equation of the main ITT impact model. 


1 One sublottery was removed from this set of analyses because all six participants in this sublottery had enrolled 
in college within 2 years after expected high school graduation (i.e., by Year 6), and therefore the relative risk ratio 
is undefined for this sublottery beyond Year 6. 
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We explored whether the EC impacts on the six primary outcomes for the study differed 
significantly by students’ gender, minority status, low-income status, and level of prior 
mathematics and ELA achievement. ?* 


Mediation Analyses 

To address RQ3, we explored whether the impacts of ECs on college enrollment and degree 
completion within 6 years after expected high school graduation (i.e., by Year 10) were 
mediated by students’ high school experiences as measured by college credit accrual during 
high school, instructional rigor, college-going culture, and student support (see Exhibit 4 for 
details about these measures). Because college credit accrual during high school implies college 
enrollment, we examined college credit accrual during high school as a potential mediator only 
for EC impact on degree completion outcomes, not for college enrollment outcomes. Given that 
data on high school experiences were collected through the student survey administered only 
to the two youngest cohorts in the study, these analyses were limited to the 1,424 students 
who were included in the survey sample. 


For a measure to explain or mediate the EC impact, three conditions must be met: (1) there must 
be a relationship between the treatment status and the mediator, (2) the mediator must be related 
to the student outcome, and (3) there should be no unmeasured confounders of the relationship 
between the mediator and the outcome (Imai, Keele, & Yamamoto, 2010; Keele, 2015; MacKinnon 
& Dwyer, 1993).*° To assess the extent to which the total effects of ECs on students’ postsecondary 
outcomes were mediated by students’ high school experiences, we first estimated the EC impacts 
on students’ high school experiences, and then estimated the effects of high school experiences on 
students’ postsecondary outcomes while controlling for treatment status and a rich set of student 
background characteristics that might be related to both their high school experiences and 
outcomes. Based on the results from these analyses, we were able to decompose the total EC 
impact on a given outcome into a direct effect and an indirect effect as mediated through students’ 
high school experiences. (See Appendix D for details of the mediation analyses.) 


Handling of Missing Data 

By design, all students in the study have data on the outcome measures based on NSC data; 
however, not all students have data on all measures of student background characteristics, and 
some student survey respondents have missing data on high school experience measures. Rates 


12 In two of the 23 lotteries, all EC students were minorities; therefore, these two lotteries were excluded from the 
analysis of differential impact by minority status. In addition, one lottery was excluded from the analysis of 
differential impact by low-income status because no students in this lottery were qualified for free or reduced- 
price lunch. 

13 The third condition is referred to as the “sequential ignorability” assumption in the causal mediation literature 
(Imai et al., 2010; Keele, 2015). To the extent that our mediation models may not include all covariates that may 
explain the relationships between the mediators and outcomes, the results from these analyses should be 
interpreted with caution. 


AMERICAN INSTITUTES FOR RESEARCH | AIR.ORG 16 


Early College, Continued Success: Longer-Term Impact of Early College High Schools 


of missing data for the student background characteristics and measures of high school 
experiences can be found in Appendix E. To address the potential selection bias that would 
result from excluding cases with missing data, we imputed missing values on the covariates 
using multiple imputation by chained equations (Raghunathan, Lepkowski, Van Hoewyk, & 
Solenberger, 2001). The multiple imputation model included all outcome measures, covariates, 
interaction terms, and mediators used in addressing the RQs, as well as indicators for treatment 
status and lotteries. We generated 10 imputed data sets, conducted all analyses using each 
imputed data set separately, and then combined estimates across the 10 data sets based on 
standard multiple imputation combination rules, which take into account the uncertainty in 
imputed values both within and across the imputed data sets (Little & Rubin, 2002). 


Results 


In this section, we present the main findings for each of the three RQs guiding the study. 
Further details about the main findings and additional findings from sensitivity analyses are 
provided in Appendices F and G, respectively. 


EC Impacts on Student Outcomes (RQ1) 

RQ1 of the study focuses on the overall impacts of ECs on students’ postsecondary outcomes. 
Below we first present findings about EC impacts on outcomes in the college enrollment domain, 
including enrollment in different types of colleges and enrollment profiles over time, and then we 
present findings related to EC impacts on outcomes in the degree attainment domain. 


ITT Impact on College Enrollment 

Our analysis showed that being admitted to an EC had a statistically significant, positive impact on 
college enrollment within 6 years after expected high school graduation (i.e., by Year 10): 84.2% of 
EC students had at least one record of college enrollment in the NSC during the time period, 
roughly 7 percentage points higher than the college enrollment rate for control students (77.0%, 
see Exhibit 5). This difference was driven primarily by the much higher rate of enrollment in 2-year 
colleges for EC students (65.8%) than for the control students (46.8%). The two groups of students 
did not significantly differ in the rate of enrolling in 4-year colleges by the end of the sixth year after 
expected high school graduation (57.6% of EC students compared with 56.7% of control students). 
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Exhibit 5. Percentage of Students Who Enrolled in College Within 6 Years After Expected High 
School Graduation, by Type of College and Study Group 


84.2%*** 
77.0% 


65.8%*** 
57.6% 56.7% 
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Notes. N = 2,458 (1,044 EC, 1,414 control). The percentages for EC students are unadjusted percentages; the 
percentages for control students were computed based on the unadjusted percentages for EC students and 
estimated EC effects. 

*#* 0 < 001. 


For both enrollment in any type of postsecondary institutions and enrollment in 2-year colleges, 
we found that winning an EC lottery had a significant impact on enrollment rates by the end of 
each year between Year 4 (i.e., the fourth year of high school) and Year 10 (i.e., 6 years after 
expected high school graduation), although the size of the impact tended to decrease over time 
(see Exhibits 6 and 7). By the end of Year 4, 63.8% of EC students and 23.0% of control students 
had enrolled in college, a difference of more than 40 percentage points. By the end of Year 5, 
this difference was reduced to about 10 percentage points, and by the end of Year 10, this 
difference was further reduced to about 7 percentage points. Similarly, the difference in 
enrollment in 2-year colleges by the end of Year 4 was almost 38 percentage points (48.9% of 
EC students compared with 11.3% of control students). This difference decreased to about 25 
percentage points by the end of Year 5, and further decreased to 19 percentage points but 
remained statistically significant by the end of Year 10. 
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Exhibit 6. Percentage of Students Who Enrolled in Any Colleges Between Year 4 and Year 10, 
by Study Group 
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Notes. N = 2,458 (1,044 EC, 1,414 control). The percentages for EC students are unadjusted percentages; the percentages 
for control students were computed based on the unadjusted percentages for EC students and estimated EC effects. 
Year x, x = 4~10, refers to x years after starting high school. 

*** 9 < 001. 


Exhibit 7. Percentage of Students Who Enrolled in 2-Year Colleges Between Year 4 and Year 
10, by Study Group 
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Notes. N = 2,458 (1,044 EC, 1,414 control). The percentages for EC students are unadjusted percentages; the percentages 
for control students were computed based on the unadjusted percentages for EC students and estimated EC effects. 
Year x, x = 4~10, refers to x years after starting high school. 

*** 9 < 001. 
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A general decline over time in the size of the EC impact is also evident for enrollment in 4-year 
colleges. As Exhibit 8 shows, by the end of Year 4 (i.e., the fourth year of high school), 22.3% of 
EC students had enrolled in 4-year colleges compared with 9.1% of control students—a 
statistically significant difference of approximately 13 percentage points. This difference 
decreased to less than 5 percentage points (which was no longer statistically significant) by the 
end of Year 5, but widened slightly to about 6 percentage points by the end of Year 6, a 
difference that achieved statistical significance.** After Year 6, the EC impact on the rate of 
enrollment in 4-year colleges further declined and was no longer statistically significant. 


Exhibit 8. Percentage of Students Who Enrolled in 4-Year Colleges Between Year 4 and Year 
10, by Study Group 
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Notes. N = 2,458 (1,044 EC, 1,414 control). The percentages for EC students are unadjusted percentages; the 
percentages for control students were computed based on the unadjusted percentages for EC students and 
estimated EC effects. 

Year x, x = 4~10, refers to x years after starting high school. 

* p< .05; *** p< .001. 


To further examine whether there was an EC impact on the type of institutions that students 
attended, we estimated EC impacts on enrollment in selective 4-year colleges as defined by 
Barron’s Profiles of American Colleges (2008, 2010, 2012, 2014, 2016, 2017, 2018). As 
illustrated in Exhibit 9, while EC students were significantly more likely than control students to 
enroll in selective 4-year colleges in the fourth year of high school (6.1% of EC students 
compared with 0.8% of control students), the two groups of students did not differ significantly 


M4 Two of the ECs in the study have 5-year programs. It is likely that many students from these two ECs enrolled in 
4-year colleges in Year 6—the first year after their expected high school graduation—rather than in Year 5 when 
they were still in high school. This may be one potential explanation for the increased EC-control gap in the rate of 
enrollment in 4-year colleges between Year 5 and Year 6. 
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in their enrollment in selective 4-year colleges after expected high school graduation. By the 
end of Year 10, approximately 18% of both EC students and control students had enrolled in 
selective 4-year colleges. 


Exhibit 9. Percentage of Students Who Enrolled in Selective 4-Year Colleges Between Year 4 
and Year 10, by Study Group 
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Notes. N = 2,458 (1,044 EC, 1,414 control). The percentages for EC students are unadjusted percentages; the percentages 
for control students were computed based on the unadjusted percentages for EC students and estimated EC effects. 
Year x, x = 4~10, refers to x years after starting high school. 

*** 9 < 001. 


To gain a more complete picture of the EC impact on students’ college enrollment, we further 
examined students’ enrollment profiles in each year between 1 year and 6 years after expected 
high school graduation (i.e., between Year 5 and Year 10), which allowed us to track students’ 
progression over time through their academic careers. In each year, students were classified 
into one of four categories: (1) had not yet enrolled in college (i.e., the baseline category), (2) 
ever enrolled exclusively in 2-year colleges, (3) ever enrolled in a 4-year college with a prior 
enrollment in a 2-year college, or (4) ever enrolled in a 4-year college without a prior 
enrollment in a 2-year college. Based on a multinomial logistic regression analysis, we 
estimated the EC impact on the relative risk for each of the last three categories in each given 
year, which is defined as the ratio of the probability of being in a particular enrollment category 
to the probability of being in the baseline category by the end of the given year. A relative risk 
that is greater than 1 indicates that students are more likely to belong in the specified 
enrollment category relative to not having enrolled in college. In contrast, a relative risk that is 
less than 1 indicates that students are less likely to belong in the specified enrollment category 
relative to not having enrolled in college. The relative risks for EC and control students based on 
this set of analyses are presented in Exhibit 10. 
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Exhibit 10. Relative Risks Associated With Each Postsecondary Enrollment Category Between Year 5 and Year 10, by Study Group 
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Notes. N = 2,452 (1,041 EC, 1,411 control). Relative risk is defined as the probability of being in a given enrollment category relative to the probability of not 
having enrolled in college (i.e., the baseline category). Relative risks for EC students were computed based on the unadjusted percentage of EC students in 

each of the enrollment categories. Relative risks for control students are adjusted relative risks that were computed based on the unadjusted relative risks for 
EC students and estimated EC effects. 


Year x, x = 5~10, refers to x years after starting high school. 


* p< .05; *** p< .001. 
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As demonstrated in the first two panels of Exhibit 10, in each year between Year 5 and Year 10, 
EC students were more likely to have “enrolled exclusively in 2-year colleges” and “enrolled in a 
4-year college with a prior enrollment in a 2-year college” than to have not enrolled in college, 
with relative risks greater than one for both enrollment categories. In contrast, the relative risks 
for both categories are less than one for the control students in each year examined, indicating 
that control students were /ess likely to be in these enrollment categories than in the baseline 
category (i.e., had not enrolled in college). These differences between EC students and control 
students in each year examined were statistically significant (p < .001). 


The third panel of Exhibit 10 shows the relative risks associated with “enrolling in a 4-year 
college without a prior enrollment in a 2-year college” in each year between Year 5 and Year 10 
for EC students and control students. For each year examined, the relative risk was lower for EC 
students than for control students, and the difference was statistically significant (jp < .05) for all 
years examined except Year 6. Taken together, results in Exhibit 10 show that EC students were 
more likely to enroll in a 4-year college after starting their education at a 2-year college, while 
control students were more likely to start at a 4-year college without a prior enrollment at a 2- 
year college. 


To facilitate the interpretation of the results about students’ postsecondary enrollment profiles, 
Exhibit 11 presents the distribution of EC students and control students across the four 
enrollment categories in two select years—1 year after expected high school graduation (Year 5) 
and 6 years after expected high school graduation (Year 10). Clearly, the two groups of students 
manifested different enrollment profiles. Mirroring our main impact estimates, Exhibit 11 
shows that the percentage of EC students who had not enrolled in college was lower than the 
percentage for control students by Year 5 (21.1% vs. 28.6%) and Year 10 (15.8% vs. 20.3%). The 
percentage of EC students who had enrolled in a 4-year college without a prior enrollment at a 
2-year college was also lower than that for control students by Year 5 (24.3% vs. 46.3%) and 
Year 10 (25.9% vs. 50.3%). The percentages of EC students in the other two enrollment 
categories (i.e., enrolled exclusively in 2-year colleges and enrolled in a 4-year college after a 
prior enrollment in a 2-year college), however, were both much higher than the percentages for 
control students by Year 5 and Year 10.*° 


15 The percentage of students classified as having “enrolled exclusively in 2-year colleges” decreased between Year 
5 and Year 10 as some students transferred from 2-year to 4-year colleges, thus being reclassified as “enrolled ina 
4-year college with a prior enrollment in a 2-year college.” 
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Exhibit 11. Percentage of Students in Each Postsecondary Enrollment Category 1 Year and 6 
Years After Expected High School Graduation, by Study Group 


EC Students Control Students EC Students Control Students 
Year 5 Year 10 
™ Enrolled in a 4-year college without prior enrollment in a 2-year college 
Enrolled in a 4-year college with prior enrollment in a 2-year college 


w Enrolled exclusively in 2-year colleges 


@ Not enrolled in college 


Notes. N = 2,452 (1,041 EC, 1,411 control). The percentages for EC students are unadjusted percentages; the 
percentages for control students were computed based on the unadjusted percentages for EC students and 
estimated EC effects. 

Year x, x = 5, 10, refers to x years after starting high school. 


In summary, our findings show that EC students were significantly more likely than control 
students to enroll in 2-year colleges, and they were also significantly more likely to transfer 
from 2-year colleges to 4-year colleges over time. Control students, in contrast, were 
significantly more likely than EC students to enroll in 4-year colleges without prior enrollment in 
2-year colleges. Taken together, we did not observe significant differences between EC and 
control students in overall enrollment in 4-year colleges over time. 


ITT Impact on Degree Attainment 

The results for the EC impact on the primary outcome measures of degree attainment are 
summarized in Exhibit 12. Within 6 years after expected high school graduation (i.e., by the end 
of Year 10), 45.4% of EC students completed postsecondary degrees, compared with 33.5% of 
control students (p < .001). This 12-percentage-point difference was largely driven by the 
difference between the two study groups in the percentage of students who completed an 
associate’s degree or certificate by the end of Year 10—29.3% of EC students compared with 
11.1% of control students, a difference exceeding 18 percentage points (p < .001). The 
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difference between the study groups in bachelor’s degree completion by the end of Year 10 
was much smaller yet still statistically significant—5.2 percentage points (30.1% of EC students 
compared with 24.9% of control students, p < .05). 


Exhibit 12. Percentage of Students Who Completed a College Degree Within 6 Years After 
Expected High School Graduation, by Type of Degree Completed and Study Group 
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Notes. N = 2,458 (1,044 EC, 1,414 control). The percentages for EC students are unadjusted percentages; the 
percentages for control students were computed based on the unadjusted percentages for EC students and 
estimated EC effects. 

* 0 < .05; *** p< .001. 


Further analyses revealed that being admitted to an EC had a significant, positive impact on 
degree completion by the end of each year between Year 4 and Year 10 (see Exhibit 13). While 
the difference in degree completion rate between the EC students and control students 
remained statistically significant over time, the size of the difference decreased from a 
maximum of 22.5 percentage points (by the end of Year 6 and Year 7) to 11.9 percentage points 
(by the end of Year 10). This indicates that, over time, control students began to “catch up” to 
EC students in terms of degree completion, but a significant difference between the two groups 
of students remained 6 years after expected high school graduation. 


AMERICAN INSTITUTES FOR RESEARCH | AIR.ORG 25 


Early College, Continued Success: Longer-Term Impact of Early College High Schools 


Exhibit 13. Percentage of Students Who Completed Any Postsecondary Degree Between Year 
4 and Year 10, by Study Group 
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Notes. N = 2,458 (1,044 EC, 1,414 control). The percentages for EC students are unadjusted percentages; the 
percentages for control students were computed based on the unadjusted percentages for EC students and 
estimated EC effects. 

Year x, x = 4~10, refers to x years after starting high school. 

*** 0 < 001. 


Regarding the completion of an associate’s degree or certificate, the results presented in 
Exhibit 14 show that the differences between EC students and control students were largely 
stable in both size and statistical significance over time. By the end of the fourth year of high 
school, 21.7% of EC students and less than 1% of control students had completed an associate’s 
degree or certificate (p < .001). EC-control differences in rates of associate’s degree or certificate 
completion hovered between 20 and 22 percentage points and remained statistically 
significant through the final year of observation, 6 years after expected high school graduation, 
when the difference reduced slightly to 18.2 percentage points (p < .001). 
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Exhibit 14. Percentage of Students Who Completed an Associate’s Degree or Certificate 
Between Year 4 and Year 10, by Study Group 
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Notes. N = 2,458 (1,044 EC, 1,414 control). The percentages for EC students are unadjusted percentages; the 
percentages for control students were computed based on the unadjusted percentages for EC students and 
estimated EC effects. 

Year x, x = 4~10, refers to x years after starting high school. 

*** 0D < 001. 


Where bachelor’s degree completion is concerned, the difference between the two study 
groups first widened and then narrowed (but remained statistically significant) over the time 
frame we examined. As shown in Exhibit 15, by the end of Year 6, 2.3% of EC students and less 
than 1% of control students had completed a bachelor’s degree, a difference of about 2 
percentage points (p < .05). By the end of Year 8, which represents the fourth year after 
expected high school graduation, 20.7% of EC students completed a bachelor’s degree, which 
was almost 10 percentage points higher than the rate for control students (10.9%; p < .001). 
This difference narrowed considerably—to 5.2 percentage points (30.1% of EC students and 
24.9% control students)—by the end of Year 10 but remained statistically significant (p < .05). 
These findings again suggest that control students began to “catch up” to EC students over time 
in terms of bachelor’s degree completion, but a significant difference between the two groups 
of students remained 6 years after expected high school graduation. 
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Exhibit 15. Percentage of Students Who Completed a Bachelor’s Degree Between Year 6 and 
Year 10, by Study Group 
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Notes. N = 2,458 (1,044 EC, 1,414 control). The percentages for EC students are unadjusted percentages; the 
percentages for control students were computed based on the unadjusted percentages for EC students and 
estimated EC effects. 

Year x, xX = 4~10, refers to x years after starting high school. 

* 0 <.05; *** p< .001. 


CATE Estimates for College Enrollment and Degree Attainment 

Because not all students who were selected for admission to an EC enrolled in the EC, ITT 
estimates of EC impacts are conservative in nature. To account for no-shows (i.e., treatment 
students who did not enroll in an EC) and crossovers (i.e., control students who somehow 
enrolled in an EC), we performed CATE analyses to supplement the ITT analyses (see Appendix 
D for technical details about the CATE analyses). Results from the CATE analyses mirror the 
results from the ITT analyses, with significant complier effects on all primary outcome measures 
in both the college enrollment domain (see Exhibit 16) and degree attainment domain (see 
Exhibit 17) except for enrollment in 4-year colleges. As expected, the CATE estimates tend to be 
larger than the ITT estimates of EC impact. For example, while the ITT estimate indicates an 
overall EC-control difference of approximately 5 percentage points in the probability of 
completing a bachelor’s degree within 6 years after expected high school graduation (i.e., by 
Year 10), the CATE estimate indicates a group difference of 8.5 percentage points among 
compliers. 
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Exhibit 16. Percentage of Students Who Enrolled in College Within 6 Years After Expected High 
School Graduation Based on CATE Estimates, by Type of College and Study Group 
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Notes. N = 2,458 (1,044 EC, 1,414 control). The percentages for EC students are unadjusted percentages; the 
percentages for control students were computed based on the unadjusted percentages for EC students and 
estimated EC effects. 

*** 0 < 001. 


Exhibit 17. Percentage of Students Who Completed a College Degree Within 6 Years After 
Expected High School Graduation Based on CATE Estimates, by Type of Degree Completed 
and Study Group 
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Notes. N = 2,458 (1,044 EC, 1,414 control). The percentages for EC students are unadjusted percentages; the 
percentages for control students were computed based on the unadjusted percentages for EC students and 
estimated EC effects. 

** 9 < 01; *** p< .001. 
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Sensitivity Analysis Results 

In addition to the ITT and CATE analyses, we performed two sets of sensitivity analyses. First, 
we re-estimated EC impacts after removing the EC site where student demographic and prior 
achievement data had to be imputed for all EC and control students because their state did not 
allow us to link student background data to outcome data. Second, we re-estimated the EC 
impacts on primary college enrollment and degree attainment outcomes with a random-effects 
model, which allows the EC impact to vary across lotteries. The results of these sensitivity 
analyses are summarized below; see Appendix G for more details on these analyses. 


Impact analyses excluding one EC with imputed background data for all students. Results 
from this set of analyses mirror the ITT results presented earlier with one exception: the EC 
impact on bachelor’s degree completion within 6 years after expected high school graduation 
(i.e., by Year 10) was slightly smaller and only marginally significant (p = .096) after removing 
one site where background data were imputed for all students. 


Impact analyses based on a random-effects model. Results from random-effects ITT analyses 
were largely consistent with the results from fixed-effects ITT analyses with one exception: the 
EC impact on bachelor’s degree completion within 6 years after expected high school 
graduation (i.e., by Year 10), which was statistically significant (p < .05) based on the fixed- 
effects model, was somewhat smaller and no longer significant (p = .159) based on a random- 
effects model. In addition, we observed significant variation in impact estimates across lotteries 
for the outcomes of completing any type of degree and completing an associate’s degree or 
certificate within 6 years after expected high school graduation (see Exhibit G-4 in Appendix G). 


Differential EC Impacts on Student Outcomes (RQ2) 

In this section, we present a summary of findings for RQ2: Did the impacts of ECs vary by 
student background characteristics? Detailed results are provided in Appendix F. Overall, we 
found that the EC impacts on college enrollment and degree attainment outcomes did not 
significantly differ by gender, race/ethnicity, or low-income status. However, we did find 
significant differential impacts based on students’ Grade 8 achievement test scores. 


Differential Impacts on College Enrollment 

We found that the EC impact on enrolling in a 2-year college within 6 years after expected high 
school graduation (i.e., by Year 10) differed significantly by students’ prior achievement in ELA 
(see Exhibit 18). The EC impact on enrolling in 2-year colleges was stronger for students with 
higher levels of Grade 8 ELA achievement than for students with lower levels of prior ELA 
achievement. The differential impact of ECs by students’ Grade 8 mathematics achievement 
was similar in magnitude but only marginally significant (p < .10). 
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Exhibit 18. Percentage of Students Who Enrolled in a 2-Year College 6 Years After Expected 
High School Graduation, by Prior Achievement Level and Study Group 


Grade 8 ELA achievement: One SD above the 
average 


Grade 8 ELA achievement: Average 


Grade 8 ELA achievement: One SD below the 
average 


Grade 8 math achievement: One SD above the 
average 


Grade 8 math achievement: Average 


Grade 8 math achievement: One SD below the 
average 


0% 20% 40% 60% 80% 100% 
Percent enrolled 


meECStudents m Control Students 


Notes. ELA = English language arts; SD = standard deviation. N = 2,458 (1,044 EC, 1,414 control). The treatment and 
control group percentages are predicted probabilities based on a differential impact model with grand-mean 
centered covariates other than prior achievement. Estimate of differential EC Impact by prior ELA achievement is 
significant at p < .05; estimate of differential EC impact by prior math achievement is marginally significant at 
p<.10. 


Differential Impacts on Degree Attainment 

We also found that the EC impact on completing an associate’s degree or certificate within 6 
years after expected high school graduation differed by students’ prior achievement in both 
ELA and mathematics (see Exhibit 19). The impact on completing an associate’s degree or 
certificate was stronger for students with higher levels of Grade 8 achievement than for 
students with lower levels of prior achievement. We did not observe significant differential 
impacts on bachelor’s degree completion. 
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Exhibit 19. Percentage of Students Who Completed an Associate’s Degree or Certificate 6 
Years After Expected High School Graduation, by Prior Achievement Level and Study Group 
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Notes. ELA = English language arts; SD = standard deviation. N = 2,458 (1,044 EC, 1,414 control). The treatment and 
control group percentages are predicted probabilities based on a differential impact model with grand-mean 
centered covariates other than prior achievement. Estimate of differential EC Impact by prior ELA achievement is 
significant at p < .01; estimate of differential EC impact by prior math achievement is significant at p < .001. 


The Mediating Role of High School Experiences (RQ3) 

To better understand EC impacts on student outcomes, we explored whether differences in 
students’ experiences during high school explained observed differences in college enrollment 
and degree attainment outcomes 6 years after expected high school graduation. In this section, 
we first demonstrate EC impacts on students’ high school experience as measured through a 
survey administered as part of the original impact study. We then present EC impacts on college 
enrollment and degree completion outcomes 6 years after expected high school graduation 
within our sample of survey respondents. Finally, for each primary outcome, we examine: (1) 
the percentage of the “total EC effect” that was explained by high school experience measures 
(i.e., indirect effect) and (2) the percentage of total EC effect not explained by high school 
experience measures (i.e., direct effect). (See Appendix D for technical details about the 
mediation analyses.) 


As shown in Exhibit 20, EC students reported significantly more positive high school experiences 
than control students. For the measures of college-going culture, student support, and 
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instructional rigor (see Exhibit 4 for details about these survey scales), the average scale scores 
for the EC students were approximately one third of a standard deviation higher than those for 
the control students. We also found that EC students were significantly more likely than control 
students to report college credit accrual during high school, which was measured as either 
earning college credit directly from college courses or passing at least one AP exam during high 
school. More than two thirds of EC students (69.3%) reported accruing college credit during 
high school, compared with about one third (35.5%) of the control students. 


Exhibit 20. Average Scores on High School Experience Measures, by Study Group 
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Notes. N = 1,424 (771 EC, 653 control). Survey scales of college-going culture, student support, and instructional 
rigor were standardized within the study sample to have a mean of O and a standard deviation (SD) of 1. The 
average scores for EC students are unadjusted score; the average scores for control students were computed 
based on the average scores for EC students and estimated EC effects. 

*** 0 < 001. 


Next, we examined EC impacts on college enrollment and degree completion outcomes among 
students who participated in the survey. Consistent with the impact findings based on the full 
sample, we found significant impacts on the college enrollment and degree completion 
outcomes for the survey participants with one exception: we did not observe a significant EC 
impact on enrollment in a 4-year college (see Exhibit F-8 in Appendix F for detailed results). 
Therefore, we did not include this outcome in mediation analyses. For the rest of the outcomes, 
the magnitude of the EC impact ranged from an effect size of 0.19 (for bachelor’s degree 
completion) to 0.81 (for completing an associate’s degree or certificate). 


The results of our mediation analyses indicate that high school experiences explained a 
significant portion (29%) of the total EC effect on enrollment in any type of institution, but did 
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not significantly explain the EC effect on enrollment in a 2-year college (see Exhibit 21). The 
direct EC effect on enrollment in a 2-year college accounted for 97% of the total EC effect and 
was statistically significant (p < .05). 


Exhibit 21. Total, Direct, and Indirect EC Effects on College Enrollment Among Students in the 
Student Survey Sample, 6 Years After Expected High School Graduation 
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Notes. N = 1,424 (771 EC, 653 control). The total effect is equal to the sum of the direct and the indirect effects. 
Direct and indirect effects were estimated with the following high school experience measures as mediators: 
instructional rigor, college-going culture, and student support. 

*p < .05; ***p < .001. 


High school experience measures appear to be stronger mediators for EC impacts on degree 
completion outcomes than for EC impacts on college enrollment outcomes. As shown in Exhibit 
22, measures of high school experience explained 59% of the total EC effect on completing any 
type of degree, 21% of the EC effect on completing an associate’s degree or certificate, and 86% 
of the EC effect on bachelor’s degree completion (p < .05 for all three indirect effects). For each 
of these outcomes, detailed results showed that college credit accrual during high school 
explained the largest percentage of the total EC effect (see Exhibit F-9 in Appendix F). After 
accounting for high school experiences, the direct EC effect was statistically significant only for 
completing an associate’s degree or certificate and no longer significant for overall degree 
completion or completing a bachelor’s degree. 


AMERICAN INSTITUTES FOR RESEARCH | AIR.ORG 34 


Early College, Continued Success: Longer-Term Impact of Early College High Schools 


Exhibit 22. Total, Direct, and Indirect EC Effects on Degree Completion Among Students in the 
Student Survey Sample, 6 Years After Expected High School Graduation 
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Notes. N = 1,424 (771 EC, 653 control). The total effect is equal to the sum of the direct and the indirect effects. 
Direct and indirect effects were estimated with the following high school experience measures as mediators: 
instructional rigor, college-going culture, student support, and college credit accrual during high school. 

*p < .05; ***p < .001. 


Summary and Discussion 


With additional years of data collected after the completion of the original EC impact study, this 
efficacy follow-up study assessed longer-term impacts of ECs on students’ college enrollment 
and degree attainment outcomes as well as differential EC impacts based on student 
background characteristics. In addition, we explored the role of students’ high school 
experiences in mediating the EC impacts on students’ college enrollment and degree 
completion 6 years after expected high school graduation. Overall, the findings show positive 
EC impacts on various postsecondary outcomes over the time period examined in this study. 
These positive impacts were similarly observed for students from different family backgrounds, 
but some of the impacts were stronger for students with higher levels of Grade 8 achievement. 
Moreover, we found that while students’ high school experiences had a limited role in 
mediating EC impacts on college enrollment outcomes, they accounted for significant portions 
of the EC impacts on degree completion outcomes. A more detailed summary of findings 
associated with each of the study’s RQs is presented below. 


AMERICAN INSTITUTES FOR RESEARCH | AIR.ORG 35 


Early College, Continued Success: Longer-Term Impact of Early College High Schools 


EC Impacts on Students’ Postsecondary Outcomes 


RQ1: Did EC students have better postsecondary outcomes (i.e., college enrollment 
and degree attainment) than control students? 

We found that EC students had a higher overall college enrollment rate and a higher 2-year 
college enrollment rate than control students by the end of each academic year between the 
fourth year of high school and 6 years after expected high school graduation. While we 
anticipated EC-control differences in college enrollment to occur during and immediately after 
high school (since ECs partner with postsecondary institutions to facilitate college credit accrual 
during high school), significant EC-control differences in enrollment in 2-year colleges persisted 
for at least 6 years after expected high school graduation. By the end of Year 10 after starting 
high school, approximately two thirds (65.8%) of EC students enrolled in 2-year colleges, 
compared with fewer than half (46.8%) of control students (p < .001). Therefore, control 
students did not catch up with EC students in terms of enrollment in 2-year colleges over the 
extended time period examined in this study. 


In contrast, we did not observe significant EC impacts on enrollment in 4-year colleges, or 
selective 4-year colleges, after the second year after expected high school graduation. Although 
we observed significant differences between EC students and control students in the rates of 
enrolling in 4-year colleges and selective 4-year colleges during high school, control students 
caught up over time such that, 6 years after expected high school graduation, similar 
percentages of EC students (57.6%) and control students (56.7%) had enrolled in 4-year 
colleges, and about the same percentage of students in both groups (17.8% for EC and 17.7% 
for control) had enrolled in selective 4-year colleges. 


Because eight of the 10 ECs in our study partnered with 2-year colleges, one might anticipate 
that EC students would be more likely to enroll in 2-year colleges and less likely to enroll in 4- 
year colleges than control students. Examination of enrollment profiles over time confirmed 
that EC students were more likely to enroll exclusively in 2-year colleges and more likely to 
transfer from 2-year colleges to 4-year colleges than control students, whereas control 
students were more likely to directly enroll in 4-year colleges than EC students. In fact, by the 
first year after expected high school graduation, almost one quarter (24.8%) of EC students and 
only 5.5% of control students had transferred from 2-year to 4-year colleges, while the 
percentage of students who had enrolled immediately in 4-year colleges was almost twice as 
high among control students (46.3%) than among EC students (24.3%). 
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We also observed that EC impacts on college enrollment translated into significant differences in 
degree completion outcomes between EC students and control students. Within 6 years after 
expected high school graduation (i.e., by Year 10), 45.4% of EC students—compared with 33.5% 
of control students—completed a postsecondary degree (p < .001). Significant impacts were also 
observed both for completion of an associate’s degree or certificate (29.3% of EC students and 
11.1% of control students) and bachelor’s degree completion (30.1% of EC students and 24.9% 
of control students) within 6 years after expected high school graduation. Of particular note is 
that we observed a significant, positive EC impact on bachelor’s degree completion 6 years after 
expected high school graduation despite the fact that we did not observe a significant impact 
on enrollment in 4-year colleges, perhaps suggesting that EC students were better prepared for 
their education at 4-year colleges, or that they were able to complete the degree faster (within 6 
years after expected high school graduation) because of their early exposure to college during 
high school. Although most of the ECs in our study partnered with 2-year colleges, it appears 
that EC students’ experiences with these colleges supported their pursuit and completion of 
bachelor’s degrees. 


Finally, we found that EC students were more likely to complete bachelor’s degrees more quickly 
than control students, which has implications for potential lifetime earnings. While the EC- 
control difference in bachelor’s degree completion rates amounted to about five percentage 
points 6 years after expected high school graduation, it was about 10 percentage points 4 years 
after expected high school graduation (by which time 20.7% of EC students and 10.9% of control 
students completed a bachelor’s degree). Therefore, while the difference in bachelor’s degree 
completion narrowed over time, the fact remains that a larger percentage of EC students 
completed bachelor’s degrees earlier in their lives, allowing them to either get a head start on 
furthering their education or entering the labor force with this credential at a younger age. 


Differential EC Impacts 


RQ2: Did the impacts of ECs vary by student background characteristics (i.e., gender, 
race/ethnicity, low-income status, and prior mathematics and ELA achievement)? 
Similar to findings from the original EC impact study (Haxton et al., 2016), we found that EC 
impacts on primary postsecondary outcomes did not vary significantly by gender, 
race/ethnicity, or low-income status, but impacts on some outcomes did vary for students with 
different levels of prior achievement. Specifically, we found that the EC impact on enrollment in 
2-year colleges was significantly stronger for students with higher levels of Grade 8 ELA 
achievement, and the EC impact on completion of an associate’s degree or certificate was 
significantly stronger for students with higher levels of Grade 8 ELA or mathematics 
achievement. One possible explanation is that, because most of the ECs in our study partnered 
with 2-year colleges, higher-achieving students in ECs might have been more likely to enroll in 
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2-year colleges (and subsequently complete an associate’s degree or certificate) during or 
immediately after high school relative to higher-achieving students in the control group who 
may have entered directly into a 4-year college after graduating from high school. We did not 
observe significant differential impacts on enrollment in 4-year colleges or completion of a 
bachelor’s degree by levels of prior achievement, indicating that starting at a 2-year college did 
not prevent higher-achieving EC students from pursuing a 4-year college degree. 


The Mediating Role of High School Experiences 


RQ3: Were the impacts of ECs mediated by students’ high school experiences (i.e., 
college credit accrual during high school, instructional rigor, college-going culture, and 
student supports)? 

Our mediation analyses revealed that students’ high school experiences accounted for a 
significant portion (29%) of the total EC impact on college enrollment, but did not significantly 
explain the EC impact on enrollment in 2-year colleges within 6 years after expected high school 
graduation. For EC impacts on degree completion outcomes, students’ high school experiences 
played a stronger mediating role, explaining 59% of the total EC impact on completing any type 
of degree, 21% of the EC impact on completing an associate’s degree or certificate, and 86% of 
the EC impact on bachelor’s degree completion 6 years after expected high school graduation. 
Among measures of students’ high school experiences, college credit accrual during high school 
explained the largest percentage of the EC impacts on degree completion outcomes, 
highlighting the importance of college credit accrual during high school for college degree 
completion. 


Caveats and Future Research 


The primary goal for this study was to assess the degree to which ECs improved students’ 
postsecondary outcomes. Such a question could theoretically be answered by randomly 
assigning students to treatment and control conditions at a randomly selected set of ECs across 
the country. Under this ideal situation, we would be able to estimate impacts that would 
generalize to over 200 ECs that currently operate nationwide. Such a study, however, is not 
feasible because many districts, schools, and parents would not allow students to be randomly 
assigned to specific high schools. In addition, to assess EC impacts on longer-term outcomes 
(e.g., bachelor’s degree completion) that are of particular interest to this study, relevant data 
would need to be collected at least 8 years after random assignment, which would be well 
beyond the timeframe for typical impact studies in education. 


Randomization of students through lottery-based admissions offered us the opportunity to 
draw valid causal conclusions about the impacts of ECs on student outcomes, including longer- 
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term outcomes, in a timely manner. The drawback, however, is that the estimated impacts only 
apply to the 10 ECs that participated in our study. Not all ECs use lotteries to determine 
admissions, and the decision about whether to use an admissions lottery is itself not random. 
To offer a lottery, a school must have more applicants than it has seats available and must 
further use a random assignment process for admissions.*® In addition, to be eligible for 
inclusion in a retrospective study such as ours, ECs that used an admissions lottery must also be 
able to provide lottery records to verify that random assignment occurred and to allow the 
study team to identify the treatment and control students among lottery participants. 
Therefore, as with any study relying on retrospective admissions lotteries, our study includes a 
nonrandom set of ECs, limiting the generalizability of study findings. 


Further, given the retrospective nature of the study, the results of our study may not apply to 
students who currently attend these ECs. Students in our sample entered Grade 9 between 
2005-06 and 2007-08, and many aspects of the ECs themselves such as staff and 
administration, partnership between the EC and the postsecondary institution, and 
admissions processes may have changed over time. In addition, changes in education policies at 
the local, state, and national levels have brought increased focus on college and career 
readiness for all students, including those who do not attend ECs. Therefore, while this study 
provides a rigorous assessment of the longer-term impacts of ECs on students who entered ECs 
over a decade ago, continued research will be necessary to determine whether positive impacts 
also occur for the current generation of EC students. 


Another limitation of the study is related to the characteristics of the student participants. 
While over half of the students in the study sample were racial minority students and almost 
half of the students were from low-income families—reflecting the ECHSI’s commitment to 
serving students from underrepresented populations—the students in our study sample were 
more academically prepared than typical students in their districts. On average, students in our 
study had Grade 8 test scores that were approximately 0.2 standard deviations above the state 
average. Therefore, it is important to note that the students in our study sample were not 
representative of the population of underrepresented students in their state. 


Because of these limitations, the study findings may not be generalizable to ECs or students 
outside of the study sample. However, the set of ECs in this study offered us the unique 
opportunity to compare the outcomes of students who were randomly selected to attend an EC 
through a lottery with the outcomes of students who did not win the lottery but would have 


16 Although one may assume that only selective, high-performing high schools would have the opportunity to 
initiate a lottery system due to oversubscription, this was not always the case. In fact, some of the ECs in this study 
implemented admission lotteries due to local or state education policies or citywide high school application 
processes. 
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otherwise attended the same EC. Thus, despite its limited external validity, this study has strong 
internal validity built upon a rigorous “gold standard” randomized experimental design. 


One avenue for future research is a focus on more recent cohorts of EC students from a more 
representative sample of ECs. Findings from such research would determine whether the 
positive EC impacts we observed within our sample of ECs apply to the larger population of ECs, 
and how these impacts may have changed over time as the characteristics of the ECs (as well as 
the opportunities available to students who do not attend ECs) have changed over time. 


Another area in need of future research is EC impacts on longer-term outcomes related to 
students’ employment and earnings. It is likely that the higher rates and earlier timing of 
degree completion for EC students would lead to better labor market outcomes (e.g., 
employment, career advancement and income) for these students in the long run. In addition, 
we hypothesize that, because EC students were more likely to receive college credit during high 
school at little or no cost to their families, EC students would be less likely to accrue student 
loan debt than control students. Until we have collected information on students’ workforce 
and financial outcomes, we can only conjecture about these longer-term impacts. 


Finally, the small number of sites in this study limited our ability to explore potential variation 
in EC impacts across sites. Future research on EC impacts that includes a larger number of sites 
could examine both variations in EC impacts and factors that may be associated with such 
variations. Such research could inform several policy- and practice-relevant questions: Which 
supports most strongly relate to EC impacts on students’ postsecondary outcomes? Can 
traditional high schools leverage current dual enrollment policies and include EC components to 
improve student access to and success in college? What is the role of state policy in the success 
of ECs? These additional lines of future research on ECs will generate valuable insights that will 
inform policies and practices pertaining to the implementation and scale-up of ECs as a 
promising dual enrollment model with proven impact on students’ postsecondary success. 
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Appendix A. Flowchart for Identifying ECs for the Original 
Impact Study 
Early Colleges open 


by fall 2007 
n= 154 


Lt 


Students in high , No students in | 
school grades High school grades 


Had graduates in . Five-year program and 
study years opened in 2007-08 


All students in 


» Not implemented for 
Early College 
y g whole school 
Used a lottery for ‘ Not oversubscribed, 
ETeliltsti eli ts or students selected non- 
randomly 
Kept lottery records , Misplaced or 


destroyed records 


Early Colleges in study 
n=10 
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Appendix B. Sample Size by School and Cohort 


Exhibit B-1. Number of Treatment and Control Students in Each Lottery and Sublottery in the 
Full Sample of the Impact Study, by Study School and Cohort 


Ninth-Grade Student Cohort 


Cohort 1 (2005-06) Cohort 2 (2006-07) Cohort 3 (2007-08) a Ke} e-) 


School A 27 43 27 43 
School B 85 103 87 54 172 157 
School C 30 282 71 231 69 257 170 770 
School D 

e Lottery 1 99 10 83 28 182 38 
e Lottery 2 7 3 2 10 9 13 
School E 62 24 62 24 
e Lottery 1 8 4 8 4 
e Lottery 2 7 7 7 7 
e Lottery 3 6 7 6 7 
School F 51 39 57 64 108 103 
School G 124 74 124 74 
School H 52 91 52 91 
School | 41 37 39 31 80 68 
School J 

e Lottery 1 3 3 3 3 
e Lottery 2 4 3 4 3 
e Lottery 3 30 9 30 9 
Total 133 343 431 551 480 520 1,044 1,414 


Notes. The total sample size across the 23 lotteries (including sublotteries) was 2,458 students (1,044 treatment 
and 1,414 control students). Shaded cells represent years that the school (1) was not yet open, (2) did not have a 
lottery, (3) did not have lottery records, or (4) did not have a graduating student cohort during the study period. 
Schools D, E, and J each had sublotteries for at least one student cohort. 
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Exhibit B-2. Number of Treatment and Control Students in Each Lottery in the Survey Sample 
of the Impact Study, by Study School and Cohort 


Ninth-Grade Student Cohort 


Cohort 2 (2006-07) Cohort 3 (2007-08) ae) ¢-)| 


Study Schools 


School B 75 93 87 54 162 147 
101 69 98 140 199 
School D 91 10 83 28 174 38 
School E i 
e Lottery 1 8 4 8 4 
e Lottery 2 -— 7 7 7 7 
e Lottery 3 6 7 6 7 
School F 39 57 64 108 103 
100 74 100 74 
31 39 31 
Total 354 317 417 336 771 653 


Notes. Total sample size across lotteries = 1,424 students (771 treatment and 653 control students). Shaded cells 
are years that the school (1) was not yet open, (2) did not have a lottery, (3) did not have lottery records, or (4) did 
not have a graduating student cohort during the study period. School E had three sublotteries for the ninth-grade 
student cohort in 2007-08. 
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Appendix C. Outcome Measures 


In this appendix, we list and define the outcome measures we examined for the efficacy follow- 
up study of EC high schools. 


Exhibit C-1. College Enrollment Outcomes 


Measure 


Description 


Primary College Enrollment Outcomes 


Enrolled in college by 
Year 10 


Enrolled in a 2-year 
college by Year 10 


Enrolled in a 4-year 
college by Year 10 


Dichotomous indicator of whether a student ever enrolled in college by the 
end of August of Year 10 after starting Grade 9 (i.e., within 6 years after 
expected high school graduation) 


Dichotomous indicator of whether a student ever enrolled in a 2-year 
college by the end of August of Year 10 after starting Grade 9 (i.e., within 6 
years after expected high school graduation) 


Dichotomous indicator of whether a student ever enrolled in a 4-year 
college by the end of August of Year 10 after starting Grade 9 (i.e., within 6 
years after expected high school graduation) 


Additional College Enrollment Outcomes 


Enrolled in college by 
Year 4 


Dichotomous indicator of whether a student ever enrolled in college by the 
end of August of Year 4 after starting Grade 9 


Enrolled in college by 
Year 5 


Dichotomous indicator of whether a student ever enrolled in college by the 
end of August of Year 5 after starting Grade 9 


Enrolled in college by 
Year 6 


Dichotomous indicator of whether a student ever enrolled in college by the 
end of August of Year 6 after starting Grade 9 


Enrolled in college by 
Year 7 


Dichotomous indicator of whether a student ever enrolled in college by the 
end of August of Year 7 after starting Grade 9 


Enrolled in college by 
Year 8 


Enrolled in college by 
Year 9 


Enrolled in a 2-year 
college by Year 4 


Enrolled in a 2-year 
college by Year 5 


Enrolled in a 2-year 
college by Year 6 


Dichotomous indicator of whether a student ever enrolled in college by the 
end of August of Year 8 after starting Grade 9 


Dichotomous indicator of whether a student ever enrolled in college by the 
end of August of Year 9 after starting Grade 9 


Dichotomous indicator of whether a student ever enrolled in a 2-year 
college by the end of August of Year 4 after starting Grade 9 


Dichotomous indicator of whether a student ever enrolled in a 2-year 
college by the end of August of Year 5 after starting Grade 9 


Dichotomous indicator of whether a student ever enrolled in a 2-year 
college by the end of August of Year 6 after starting Grade 9 
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Measure 


Enrolled in a 2-year 
college by Year 7 


Enrolled in a 2-year 
college by Year 8 


Enrolled in a 2-year 
college by Year 9 


Enrolled in a 4-year 
college by Year 4 
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Description 


Dichotomous indicator of whether a student ever enrolled in a 2-year 
college by the end of August of Year 7 after starting Grade 9 


Dichotomous indicator of whether a student ever enrolled in a 2-year 
college by the end of August of Year 8 after starting Grade 9 


Dichotomous indicator of whether a student ever enrolled in a 2-year 
college by the end of August of Year 9 after starting Grade 9 


Dichotomous indicator of whether a student ever enrolled in a 4-year 
college by the end of August of Year 4 after starting Grade 9 


Enrolled in a 4-year 
college by Year 5 


Dichotomous indicator of whether a student ever enrolled in a 4-year 
college by the end of August of Year 5 after starting Grade 9 


Enrolled in a 4-year 
college by Year 6 


Dichotomous indicator of whether a student ever enrolled in a 4-year 
college by the end of August of Year 6 after starting Grade 9 


Enrolled in a 4-year 
college by Year 7 


Dichotomous indicator of whether a student ever enrolled in a 4-year 
college by the end of August of Year 7 after starting Grade 9 


Enrolled in a 4-year 
college by Year 8 


Enrolled in a 4-year 
college by Year 9 


Enrolled in a selective 4- 
year college by Year 4 


Enrolled in a selective 4- 
year college by Year 5 


Dichotomous indicator of whether a student ever enrolled in a 4-year 
college by the end of August of Year 8 after starting Grade 9 


Dichotomous indicator of whether a student ever enrolled in a 4-year 
college by the end of August of Year 9 after starting Grade 9 


Dichotomous indicator of whether a student ever enrolled in a selective 4- 
year college (as defined by Barron’s Profiles of American Colleges) by the 
end of August of Year 4 after starting Grade 9 


Dichotomous indicator of whether a student ever enrolled in a selective 4- 
year college (as defined by Barron’s Profiles of American Colleges) by the 
end of August of Year 5 after starting Grade 9 


Enrolled in a selective 4- 
year college by Year 6 


Dichotomous indicator of whether a student ever enrolled in a selective 4- 
year college (as defined by Barron’s Profiles of American Colleges) by the 
end of August of Year 6 after starting Grade 9 


Enrolled in a selective 4- 
year college by Year 7 


Dichotomous indicator of whether a student ever enrolled in a selective 4- 
year college (as defined by Barron’s Profiles of American Colleges) by the 
end of August of Year 7 after starting Grade 9 


Enrolled in a selective 4- 
year college by Year 8 


Dichotomous indicator of whether a student ever enrolled in a selective 4- 
year college (as defined by Barron’s Profiles of American Colleges) by the 
end of August of Year 8 after starting Grade 9 


Enrolled in a selective 4- 
year college by Year 9 


Dichotomous indicator of whether a student ever enrolled in a selective 4- 
year college (as defined by Barron’s Profiles of American Colleges) by the 
end of August of Year 9 after starting Grade 9 
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Enrolled in a selective 4- 
year college by Year 10 
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Description 


Dichotomous indicator of whether a student ever enrolled in a selective 4- 
year college (as defined by Barron’s Profiles of American Colleges) by the 
end of August of Year 10 after starting Grade 9 


Enrollment profile by 
Year 5 


Enrollment profile by 
Year 6 


Categorical variable indicating whether a student had (1) not enrolled in 
college, (2) ever enrolled exclusively in 2-year colleges, (3) ever enrolled ina 
4-year college with a prior enrollment in a 2-year college, or (4) ever 
enrolled in a 4-year college without a prior enrollment in a 2-year college by 
the end of August of Year 5 after starting Grade 9 


Categorical variable indicating whether a student had (1) not enrolled in 
college, (2) ever enrolled exclusively in 2-year colleges, (3) ever enrolled in a 
4-year college with a prior enrollment in a 2-year college, or (4) ever 
enrolled in a 4-year college without a prior enrollment in a 2-year college by 
the end of August of Year 6 after starting Grade 9 


Enrollment profile by 
Year 7 


Enrollment profile by 
Year 8 


Enrollment profile by 
Year 9 


Categorical variable indicating whether a student had (1) not enrolled in 
college, (2) ever enrolled exclusively in 2-year colleges, (3) ever enrolled ina 
4-year college with a prior enrollment in a 2-year college, or (4) ever 
enrolled in a 4-year college without a prior enrollment in a 2-year college by 
the end of August of Year 7 after starting Grade 9 


Categorical variable indicating whether a student had (1) not enrolled in 
college, (2) ever enrolled exclusively in 2-year colleges, (3) ever enrolled ina 
4-year college with a prior enrollment in a 2-year college, or (4) ever 
enrolled in a 4-year college without a prior enrollment in a 2-year college by 
the end of August of Year 8 after starting Grade 9 


Categorical variable indicating whether a student had (1) not enrolled in 
college, (2) ever enrolled exclusively in 2-year colleges, (3) ever enrolled ina 
4-year college with a prior enrollment in a 2-year college, or (4) ever 
enrolled in a 4-year college without a prior enrollment in a 2-year college by 
the end of August of Year 9 after starting Grade 9 


Enrollment profile by 
Year 10 
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Exhibit C-2. Degree Attainment Outcomes 


Measure 


Description 


Primary Degree Attainment Outcomes 


Completed any 
postsecondary degree 
by Year 10 


Completed an 
associate’s degree or 
certificate by Year 10 


Dichotomous indicator of whether a student ever received any type of 
postsecondary credential, including a certificate, an associate’s degree, ora 
bachelor’s degree, by the end of August of Year 10 after starting Grade 9 
(i.e., within 6 years after expected high school graduation) 


Dichotomous indicator of whether a student ever received an associate’s 
degree or a postsecondary certificate by the end of August of Year 10 after 
starting Grade 9 (i.e., within 6 years after expected high school graduation) 


Completed a bachelor’s 
degree by Year 10 


Dichotomous indicator of whether a student ever received a bachelor’s 
degree by the end of August of Year 10 after starting Grade 9 (i.e., within 

6 years after expected high school graduation, or 150% of expected time to 
completion) 


Additional Degree Attainment Outcomes 


Completed a 
postsecondary degree 
by Year 4 


Dichotomous indicator of whether a student received any postsecondary 
degree by the end of August of Year 4 after starting Grade 9 


Completed a 
postsecondary degree 
by Year 5 


Dichotomous indicator of whether a student received any postsecondary 
degree by the end of August of Year 5 after starting Grade 9 


Completed a 
postsecondary degree 
by Year 6 


Completed a 
postsecondary degree 
by Year 7 


Completed a 
postsecondary degree 
by Year 8 


Completed a 
postsecondary degree 
by Year 9 


Completed an 
associate’s degree or 
certificate by Year 4 


Dichotomous indicator of whether a student received any postsecondary 
degree by the end of August of Year 6 after starting Grade 9 


Dichotomous indicator of whether a student received any postsecondary 
degree by the end of August of Year 7 after starting Grade 9 


Dichotomous indicator of whether a student received any postsecondary 
degree by the end of August of Year 8 after starting Grade 9 


Dichotomous indicator of whether a student received any postsecondary 
degree by the end of August of Year 9 after starting Grade 9 


Dichotomous indicator of whether a student received an associate’s degree 
or a postsecondary certificate by the end of August of Year 4 after starting 
Grade 9 
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Description 


Dichotomous indicator of whether a student received an associate’s degree 
or a postsecondary certificate by the end of August of Year 5 after starting 
Grade 9 


Completed an 
associate’s degree or 
certificate by Year 6 


Dichotomous indicator of whether a student received an associate’s degree 
or a postsecondary certificate by the end of August of Year 6 after starting 
Grade 9 


Completed an 
associate’s degree or 
certificate by Year 7 


Completed an 
associate’s degree or 
certificate by Year 8 


Completed an 
associate’s degree or 
certificate by Year 9 


Completed a bachelor’s 


degree by Year 6 


Completed a bachelor’s 


degree by Year 7 


Completed a bachelor’s 


degree by Year 8 


Dichotomous indicator of whether a student received an associate’s degree 
or a postsecondary certificate by the end of August of Year 7 after starting 
Grade 9 (i.e., within 3 years after expected high school graduation, or 150% 
of expected time to completion) 


Dichotomous indicator of whether a student received an associate’s degree 
or a postsecondary certificate by the end of August of Year 8 after starting 
Grade 9 


Dichotomous indicator of whether a student received an associate’s degree 
or a postsecondary certificate by the end of August of Year 9 after starting 
Grade 9 


Dichotomous indicator of whether a student received a bachelor’s degree by 
the end of August of Year 6 after starting Grade 9 


Dichotomous indicator of whether a student received a bachelor’s degree by 
the end of August of Year 7 after starting Grade 9 


Dichotomous indicator of whether a student received a bachelor’s degree by 
the end of August of Year 8 after starting Grade 9 


Completed a bachelor’s 


degree by Year 9 


Dichotomous indicator of whether a student received a bachelor’s degree by 
the end of August of Year 9 after starting Grade 9 


AMERICAN INSTITUTES FOR RESEARCH | AIR.ORG 


54 


Early College, Continued Success: Longer-Term Impact of Early College High Schools 


Appendix D. Technical Details on Analytic Methods 


In this appendix, we provide a detailed description of the ITT, CATE, and mediation analyses we 
performed for the efficacy follow-up study of EC high schools. 


ITT Analyses 

Our main impact analyses are ITT analyses, which estimate the impact of being offered 
admission to an EC through a lottery, regardless of whether the student actually enrolled in the 
EC. To estimate the overall ITT effects on binary postsecondary outcomes across lotteries, we 
constructed a two-level model that takes into account the clustering of students within 
lotteries. The treatment indicator was group-mean centered at the student level to make sure 
the comparisons of EC students and control students were made within rather than across 
lotteries, and thus produced unbiased estimates (Enders & Tofighi, 2007; Raudenbush, 1989). 
We modeled the intercept as a random effect to take into account the clustering of student 
outcomes within lotteries. We modeled the treatment effect as fixed at the lottery level 
because the number of lotteries in the study was too small to generate stable estimates of the 
variation in treatment effects across lotteries. 


Compared with a random-effects model with both a random intercept and a random treatment 
slope at the lottery level, our fixed-effects model is associated with greater statistical power, 
but it does not allow us to generalize study findings to EC admission lotteries beyond those in 
the study sample or to examine the variation in EC impact across lotteries. An alternative 
specification of the fixed-effects model is a student-level regression with a treatment indicator 
and lottery fixed effects. Such a model, however, suffers from the “complete separation” 
problem for certain binary outcomes in our study because, in some lotteries, 100% of the 
students experienced the outcome, or 100% of the students did not experience the outcome, 
which would lead to the exclusion of such lotteries from the impact estimation. 


Below is the specification of the random intercept, fixed-slope hierarchical generalized linear 
model that we used to assess the EC impacts on binary postsecondary outcomes: 
Level-1 Model (Student Level): 

logldy/(1 — di)] = Boj + Payj* ECy + Paj*Xj + Lin=2(Bamj * SUBLOT jj) (1) 


where 


e gy is the probability of experiencing the outcome (e.g., enrolling in college by the end of 
Year 10) for student jin lottery j; 
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e ECjisa dummy indicator for treatment status (coded 1 if the student won the EC lottery 
and 0 otherwise, centered on the lottery mean); 


e Xjis a vector of student characteristics, grand-mean centered; 


e SUBLOT mj is a set of effect-coded indicators for the m sublotteries within a lottery with 
multiple sublotteries. 1” 


e fois the average outcome (in logits) among control students in lottery j; 


e fis the difference in the average outcome between EC students and control students in 
lottery j; 


e zis the relationship between student characteristic X and the outcome in lottery j; 
e 33m is the difference in the average outcome between control students in sublottery m and 


control students across all sublotteries in the given lottery with sublotteries. 


Level-2 Model (Lottery Level): 


Boj= Yoo + ugj (2) 

Ay = Yo (3) 

$2 = 20 (4) 

Bamj = Y3mo (5) 
where 


e yoois the average outcome (in logits) among control students across all lotteries; 


e ois the average difference in the outcome between EC students and control students 
across all lotteries; 


e 20 is the average relationship between student characteristic X and the outcome across all 
lotteries; 


e Y3mo is the difference between the average outcome for control students in sublottery m 
and the average outcome for control students across all sublotteries in the given lottery 
with sublotteries; and 


@ Uo; isa random error associated with lottery j. 


17 For a given lottery with m sublotteries, SUBLOTmy was coded -1 for students in the omitted reference sublottery 
(i.e., if m= 1), 1 for students in sublottery m within the given lottery, and 0 for all other students. Given the effect 
coding, the treatment effect for such a lottery represents the equally weighted effect across the m sublotteries 
within the lottery. There is one set of sublottery indicators for each lottery with sublotteries in the level-1 
equation, although only one set is shown for simplicity. 
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The estimate of primary interest from the model is io at the lottery level, which represents a 
precision-weighted overall treatment effect across all lotteries in the study sample, with larger 
weights for lotteries with more students. For these models, the treatment effect was estimated 
in the logged odds ratio (logit) metric, where odds were defined as the probability of 
experiencing the outcome (e.g., enrolling in a 4-year college) divided by the probability of not 
experiencing the outcome. To facilitate the interpretation of the size of the treatment effects 
on binary outcomes, we converted the effect estimates into effect sizes using the Cox index 
(i.e., logged odds ratio divided by 1.65). 


To test the robustness of our main impact results based on the fixed-effects models described 
above, we also performed ITT analyses using a two-level random-effects model as sensitivity 
analyses for the three primary outcomes in each of the two outcome domains (college 
enrollment and degree attainment). The specification of the random-effects model is similar to 
that of the fixed-effects model. The only difference between the two models is that the 
outcome difference between EC students and control students in each lottery (i.e., 2, ; in 
Equation [1]) is modeled as a random effect with a random error (u,;) at the lottery level in the 
random-effects model—as shown in Equation (6) below—rather than fixed to its grand mean as 
in the fixed-effects model: 


Bij = V10 + U4; (6) 


The level-2 coefficient y\o from the above equation captures the overall treatment effect across 
all lotteries in the study, under the assumption that the lotteries in the study represent a larger 
population of EC admission lotteries. 


For the categorical outcome of postsecondary enrollment profile, we estimated multinomial 
logistic regression models where the outcome of postsecondary enrollment profile in a given 
year has four possible values/categories: (1) had not enrolled in college (i.e., the baseline or 
reference category), (2) ever enrolled exclusively in 2-year colleges, (3) ever enrolled in a 4-year 
college with prior enrollment in a 2-year college, or (4) ever enrolled in a 4-year college without 
a prior enrollment in a 2-year college. The multinomial logistic regression model, which is a 
student-level regression with a treatment indicator, lottery (and sublottery) fixed effects, and 
controls for student background characteristics, 1° is specified as follows: 


log oo = YE_, BoelOTy; + BiEC,; + BoX;,i = 2, 3,4 (7) 


18 These analyses were based on a single-level multinomial logistic regression model rather than a two-level model 
due to limitations in available software programs. As mentioned earlier, our main impact analyses were based on a 


two-level fixed-effects model rather than on a student-level model with lottery fixed effects in order to avoid the 
separation issue. For outcome measures without the separation issue, estimates based on the two alternative fixed- 
effects models were virtually identical. 
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where 
e log (=) is the log odds of being in category j versus the baseline category (i.e., having not 
m1 


enrolled in college by the given year) for student /; 
e LOT is set of dummy indicators for the k lotteries (and sublotteries) in the study sample; 
e EC;isa dummy indicator for treatment status for student i, and 


e X; represents a vector of background characteristics for student i. 


The primary coefficient of interest from the above model is £1, which is the pooled within- 
lottery difference between EC and control students in the log odds of being in category j versus 
not having enrolled in college by the end of a given year. One sublottery (containing six 
students) was removed from this set of analyses because, within 2 years after expected high 
school graduation (i.e., by Year 6), all students in this sublottery had enrolled in college, and 
therefore relative risk ratios could not be estimated for students in this sublottery. 


CATE Analyses 


To take noncompliance into account, we supplemented the ITT analyses with CATE analyses for 
the three primary outcomes in each of the two outcome domains (i.e., college enrollment and 
degree attainment). While the ITT estimates capture the effects of being offered admission to 
an EC through a lottery, the CATE estimates capture the effects of actually attending an EC for 
students who complied with their treatment assignment (i.e., compliers) in the presence of 
noncompliance (i.e., no-shows and crossovers). 


Across the 23 lotteries (including sublotteries) included in this study, no-shows (i.e., treatment 
students who did not attend the EC to which they were admitted through a lottery during the 
first year of high school) were fairly widespread and occurred in 18 lotteries, with an overall no- 
show rate of 20.9% among treatment students (see Exhibit D-1). Crossovers (i.e., control 
students who attended an EC during the first year of high school), however, occurred in only 
three lotteries, with an overall crossover rate of 2.0% among control students. The EC with the 
highest no-show rate (about 50% for all three cohorts) was in a district that ran districtwide 
lotteries for all schools where each student could participate in multiple school-specific 
lotteries. Some of the winners of the EC lottery in this district might have also won the lottery 
of another school that they preferred to attend. The lottery with the highest crossover rate 
(28.4%) was held by an EC located near another EC that was not part of the study. We classified 
control students who attended the non-study EC as crossovers because their high school 
experiences were similar to those of the treatment students. 
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Exhibit D-1. No-Show and Crossover Rates, by Study School and Cohort 


Ninth-Grade Student Cohort 


Cohort 1 (2005-06) Cohort 2 (2006-07) Cohort 3 (2007-08) 


School/Lottery | No-Shows | Crossovers | No-Shows | Crossovers | No-Shows | Crossovers 


School A 


School D 


e Lottery 1 


e Lottery 2 


School E 


e Lottery 1 


e Lottery 2 


e Lottery 3 


School F 


School J 


e Lottery 1 


e Lottery 2 


e Lottery 3 


Note. No-show and crossover are defined based on whether a student attended an EC during the first year of high 
school. Rates were computed based on 2,032 students with the enrollment data needed. School G is not included 
in this exhibit because we were not able to obtain administrative data for this site. Shaded cells represent years 
that the school (1) was not yet open, (2) did not have a lottery, (3) did not have lottery records, or (4) did not have 
a graduating student cohort during the study period. Schools D, E, and J each had multiple lotteries for at least one 
student cohort. 


We conducted the CATE analyses using the instrumental variable approach, which is particularly 
well suited for complier effect analysis in an experimental context (Angrist, Imbens, & Rubin, 
1996; Gennetian, Morris, Bos, & Bloom, 2005; Schochet & Chiang, 2009). For our study, the 
lottery-based random assignment was a natural choice of instrument for EC attendance 
because it is expected to have a positive effect on EC attendance (as lottery winners were more 
likely to attend ECs), and EC attendance is likely to be the only plausible path through which 
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winning an EC admission lottery could affect student outcomes. Moreover, it is unlikely that any 
EC applicant would deliberately do the opposite of the assignment received, regardless of the 
assignment. Thus, all three key assumptions under which the instrumental variable approach 
produces an unbiased CATE estimate (i.e., nonzero causal effect of treatment assignment on 
treatment receipt, exclusion restriction, and monotonicity [no defiers]) are likely to hold in this 
study. 


Specifically, our instrumental variable analyses involved two stages. In the first stage, we 
obtained the predicted probability of attending an EC for each student based on a two-level 
model similar to the ITT impact model, with the dependent variable being a binary measure of 
EC attendance during the first year of high school. In the second stage, the predicted probability 
of EC attendance obtained from the first stage was used to predict the student outcome. The 
second-stage equation also was similar to the ITT impact model, except that the treatment 
indicator was replaced with the predicted probability of EC attendance. The second-stage 
coefficient for the predicted probability of EC attendance captures the EC effect on compliers. 
The CATE analysis does not take into account uncertainty in the stage 1 model when estimating 
the CATE in the stage 2 model, so standard errors may be somewhat underestimated. 1° 


Mediation Analyses 

Analyses that address RQ3 explored the mechanisms through which ECs affected student 
outcomes. Specifically, we examined the extent to which the EC impacts on postsecondary 
outcomes 6 years after expected high school graduation were mediated by students’ high 
school experiences as captured by survey-based measures of college credit accrual during high 
school, instructional rigor, college-going culture, and student support. 


The mediation analyses for degree attainment outcomes included all four measures of high 
school experiences as potential mediators, as illustrated in Exhibit D-2. Given that college credit 
accrual during high school implies college enrollment, we did not include college credit accrual 
during high school as a potential mediator in the mediation analyses for college enrollment 
outcomes. Other than the exclusion of this mediator, the model illustrated in Exhibit D-2 also 
applies to mediation analyses for college enrollment outcomes. 


19 We also estimated an alternative model using the ivreg2 command in Stata. By simultaneously performing the 
stage 1 and stage 2 analyses, the ivreg2 command overcomes the limitation of incorrectly estimating standard 
errors. However, ivreg2 uses a single-level linear probability model to estimate both the probability of attending an 
EC (stage 1) and the impact on student outcomes (stage 2). The single-level model included lottery fixed effects to 
account for the clustering of students in lotteries. Results of these alternative analyses resemble the results of the 
CATE analyses presented in this report with one exception: the EC impact on bachelor’s degree completion within 
6 years after expected high school graduation (i.e., by Year 10) was only marginally significant (p = .060) using the 
ivreg2 command. 
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Exhibit D-2. EC Impacts on Degree Completion Outcomes With and Without Mediators 


Model without mediators (total effect): 


: c 
Treatment (EC) Student Outcome (Y) 


Model with mediators (indirect effects + direct effect): 


College Credit Accrual 


During High School (M1) 


al bl 


Instructional Rigor (M2) 
Treatment Le —{ 3 Student 
College-Going Culture (M3) 


a4 b4 


Student Supports (M4) 


The various paths representing the total effect (c), indirect effects (a1*b1, a2*b2, a3*b3, and 
a4*b,), and direct effect (c’) in the mediation model were estimated with the models described 
below, all of which control for lottery fixed effects (LOT;:) and student background 
characteristics (X;).2° 


Model for Estimating the Total Effect of ECs on a Binary Student Outcome: 


log (7) = Litt: BoxLOT;; + CEC; + BoX; - 


20 While our main impact analyses were based on a two-level model, our mediation analyses were based ona 
single-level model because the Stata program that we used for these analyses (binary_mediation) cannot 
accommodate multilevel models. As is the case with our two-level ITT impact model, our single-level mediation 
model also takes into account the clustering of students within lotteries and is also based on a fixed-effects rather 
than a random-effects approach. 


AMERICAN INSTITUTES FOR RESEARCH | AIR.ORG 61 


Early College, Continued Success: Longer-Term Impact of Early College High Schools 


The above model estimates the total effect of ECs on a binary student outcome, which is 
captured by coefficient c. 


Model for Estimating EC Effects on Mediators”? 


log (Pat) = YVi=1 BoxLOTp: + AEC; + B2X; (9) 
M2; = Lies BoxLOT ie + A2EC; + B2Xi + % (10) 
M3, = Y2*, BoxLOT,; + 43EC, + BX; +1; (11) 
M4, = Y34, Bor OT: + GEC; + BoX; +1; (12) 


Equations 9 through 12 estimate the EC effects on the four mediators, respectively, using a 
model similar to the main impact model. The EC effect on the binary mediator (M1) was 
estimated using a logistic regression, and the EC effects on the three continuous mediators 
(M2, M3, and M4) were estimated using ordinary least squares regressions. This set of effects is 
captured by the coefficients a1, a2, a3, and a4, respectively. 


Model for Estimating the Direct Effect of ECs and Mediator Effects on a Binary Student 
Outcome 


log (? 


J = Dies BoxLOT,; + c'EC; + Lina bmMm + B2Xi (13) 


In addition to the direct EC effect on the student outcome (i.e., c’, the EC effect accounting for 
students’ high school experiences), Equation 13 also estimates the effects of the four mediators 
on the student outcome, controlling for treatment status. This set of mediator effects is 
captured by coefficients b1, b2, b3, and b4, respectively. 


To conduct the mediation analyses, we used the Stata program binary_mediation (Enders, 
2011), which is capable of estimating indirect, direct, and total effects for models with a 
mixture of binary and continuous mediators and binary outcomes. Based on the estimates of 
the direct EC effect (c’) and the total EC effect (c), we calculated the percentage of the total EC 
effect that is explained by the high school experience measures. In addition, we calculated the 
percentage of the total EC effect that is explained by each of the mediators based on the 
indirect effect (i.e., a*b) associated with each mediator. Because our mediation model includes 
a binary mediator (M1) and three continuous mediators (W2, M3, and M4), the different path 
coefficients (i.e., a’s and b’s) and the total and direct effects (c and c’) are not on a common 


21 To keep this set of equations simple, subscripts are not used to distinguish the coefficients in the different 
equations. Thus, the coefficients shown with the same notation in different equations (e.g., Bo, and Bz) represent 
different parameter estimates in equations associated with different mediators. 
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scale. Therefore, to examine the magnitude of indirect effects relative to the total or direct 
effect of ECs, the different path coefficients as well as the total EC effect and direct EC effect 
were rescaled using the standardization method developed by MacKinnon and Dwyer (1993). 


Following the recommendation by Preacher and Hayes (2008), we tested the significance of the 
indirect effect associated with each mediator and the total indirect effect across all four 
mediators using bias-corrected, bootstrap confidence intervals based on 5,000 resamples. 
Compared with alternative approaches to testing the significance of indirect effects (e.g., the 
product-of-coefficient approach), bootstrapping, as a nonparametric resampling procedure, 
does not impose the assumption of normality of the sampling distribution and is considered the 
most powerful and reasonable method of obtaining confidence intervals for indirect effects 
under most conditions (Briggs, 2006; Preacher & Hayes, 2008; Williams, 2004; Williams & 
MacKinnon, 2008). 
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Appendix E. Missing Data 


In this appendix, we present the percentage of EC and control students with missing data on 
student background characteristics and high school experience measures. For impact and 
mediation analyses, we performed multiple imputation to account for missing data, and we 
conducted analyses on the resulting 10 imputed data sets. 


Exhibit E-1. Percentage of Students With Missing Data on Background Characteristics Prior to 
Imputation, Overall and by Study Group 


Overall EC Group Control Group 
Student Characteristic (n = 2,458) (n= 1,044) (n= 1,414) 


First-generation college-going 
Prior achievement in English language arts 
Prior achievement in mathematics 


Exhibit E-2. Percentage of Students With Missing Data on High School Experience Measures 


Prior to Imputation, Overall and by Study Group 


Overall EC Group Control Group 
High School Experience Measures (n= 1,424) (n =771) (n = 653) 


College credit accrual during high school 11.7% 14.7% 
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Appendix F. Summary of Study Findings 


In this appendix, we provide detailed findings from the impact, CATE, differential impact, and 
mediation analyses performed for the efficacy follow-up study of EC high schools. 


Impact Estimates 


Exhibit F-1. ITT Estimates of the Overall EC Impacts on College Enrollment Outcomes 


Effect Control 
in Standard| EC Group Group 
Outcome Logits Error Probability | Probability | Difference p-Value 


0.464) 0.121 84.2% 77.0% 7.2% 0.281 | .000 
0.781 | 0.108 65.8% 46.8% 19.0% 0.474 | .000 


0.035 | 0.106 57.6% 56.7% 0.8% 0.021 | .744 


Selective 4-year college 0.008 | 0.160 17.8% 17.7% 0.1% 0.005 |} .959 
enrollment by Year 10 


College enrollment by Year 4 1.775 | 0.115 63.8% 23.0% 40.8% 1.076 | .000 


College enrollment by Year 5 0.509 | 0.112 78.9% 69.2% 9.7% 0.308 | .000 
College enrollment by Year 6 0.520 | 0.116 81.8% 72.8% 9.0% 0.315 | .000 


College enrollment by Year 7 0.465 | 0.118 82.6% 74.9% 7.7% 0.282 | .000 


College enrollment by Year 8 0.478 | 0.120 83.3% 75.6% 7.7% 0.290 | .000 
College enrollment by Year 9 0.481 | 0.121 83.8% 76.2% 7.6% 0.292 | .000 


2-year college enrollment by 1.775 | 0.115 48.9% 11.3% 37.6% 1.222 | .000 
Year 4 


2-year college enrollment by 0.509 | 0.112 57.0% 31.9% 25.1% 0.630 |} .000 
Year 5 


2-year college enrollment by 0.520 | 0.116 61.3% 39.1% 22.2% 0.548 | .000 
Year 6 


0.465 | 0.118 62.9% 43.2% 19.7% 0.487 | .000 


0.478 | 0.120 64.2% 45.0% 19.2% 0.475 | .000 


2-year college enrollment by 0.481 | 0.121 64.9% 46.0% 19.0% 0.472 | .000 
Year 9 
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Effect Control 
in Standard | EC Group chee} ty) Effect 
Outcome Logits tdce)g Probability | Probability | Difference | Size | p-Value 


1.060 | 0.172 22.3% 9.1% 13.3% 0.642 | .000 
Year 4 

0.167 | 0.109 49.2% 45.1% 4.1% 0.101 | .128 
Year 5 


4-year college enrollment by 0.237 | 0.107 53.4% 47.5% 5.9% 0.144 | .026 
Year 6 


4-year college enrollment by 0.137 | 0.106 55.2% 51.8% 3.4% 0.083 |} .196 
Year 7 


4-year college enrollment by 0.109 | 0.106 55.9% 53.2% 2.7% 0.066 | .300 
Year 8 


4-year college enrollment by 0.079 | 0.106 56.9% 54.9% 1.9% 0.048 | .455 
Year 9 


Selective 4-year college 2.043 | 0.467 6.1% 0.8% 5.3% 1.238 | .000 
enrollment by Year 4 


oO 0.199 | 12.6% | 11.4% | 1.1% |0.065| .592 
ee oes 0.193 | 13.6% | 11.8% | 1.8% | 0.098) .400 


Selective 4-year college 0.049 | 0.180 14.6% 14.0% 0.6% 0.030 | .783 
enrollment by Year 7 


Selective 4-year college 0.042 | 0.167 16.6% 16.0% 0.6% 0.026 | .800 
enrollment by Year 8 


Selective 4-year college 0.027 | 0.162 17.3% 17.0% 0.4% 0.017 | .866 
enrollment by Year 9 


Notes. N = 2,458 (1,044 EC, 1,414 control). The EC group percentages are unadjusted percentages; the control 
group percentages were computed based on the unadjusted EC group percentages and estimated EC effects. We 
computed the effect sizes for these binary outcomes by dividing the logged odds ratio by 1.65 (i.e., the Cox index), 
as recommended by the WWC. 

Year x, x = 4~10, refers to x years after starting high school. 
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Exhibit F-2. ITT Estimates of the Overall EC Impacts on Enrollment Profiles 


Treatment Control 
Effectin | Standard Group (clgelt) Relative Risk 


Outcome Logits tagelg Relative Risk | Relative Risk thao) p-Value 


Enrollment Exclusively in 2-Year Colleges (relative to not enrolled in college) 


Year 5 0.719 0.131 1.414 0.688 2.053 .000 


Year 6 0.671 0.136 1.568 0.802 1.956 .000 
Year 7 0.637 0.139 1.571 0.831 1.892 .000 


Year 8 0.671 0.140 1.644 0.841 1.955 .000 
Year 9 0.702 0.142 1.663 0.824 2.018 .000 


Year 10 0.715 0.143 1.685 0.824 2.045 .000 


Enrollment in a 4-Year College With a Prior Enrollment in a 2-Year College (relative to not enrolled in college) 
Year 5 1.816 0.183 1.177 0.192 6.148 .000 
Year 6 1.688 0.171 1.547 0.286 5.407 .000 


Year 7 1.378 0.161 1.703 0.430 3.965 .000 


1.306 0.159 1.816 0.492 3.692 .000 


1.230 0.157 1.923 0.562 3.420 .000 


1.168 0.156 2.006 0.624 3.215 .000 


Enrollment in a 4-Year College Without a Prior Enrollment in a 2-Year College (relative to not enrolled in 
college) 


Year 5 -0.337 0.156 1.155 1.618 0.714 .030 


Year 6 -0.271 0.157 1.379 1.808 0.763 .084 
Year 7 -0.356 0.159 1.462 2.086 0.700 .026 
Year 8 -0.356 0.160 1.540 2.198 0.701 .027 


Year 9 -0.383 0.162 1.592 2.335 0.682 .018 


Year 10 -0.414 0.163 1.636 2.477 0.661 .011 


Notes. N = 2,452 (1,041 treatment, 1,411 control). Relative risks for EC students were computed based on the 
unadjusted percentage of EC students in each of the enrollment categories in a given year. Relative risks for 
control students are adjusted relative risks that were computed based on the unadjusted relative risks for EC 
students and estimated EC effects. The relative risk ratio for each enrollment profile outcome is the ratio of the EC 
relative risk to the control group relative risk. 

Year x, x = 5~10, refers to x years after starting high school. 
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Exhibit F-3. ITT Estimates of the Overall EC Impacts on Degree Attainment Outcomes 


Control 


Effectin | Standard | EC Group (chgelt) Effect 
Outcome Logits tage) Probability | Probability | Difference | Size | p-Value 


Completed any degree by 0.503 11.9% |0.305| .000 
Year 10 


Completed an associate’s 18.2% |0.725| .000 
degree or certificate by 
Year 10 


Completed a bachelor’s 5.2% 0.157} .037 
degree by Year 10 


Completed any degree by 20.9% |/2.116; .000 
Year 4 


Completed any degree by 22.3% |1.893] .000 
Year 5 


Completed any degree by 22.5% /1.546; .000 
Year 6 


Completed any degree by 22.5% |1.178; .000 
Year 7 


Completed any degree by 19.8% |0.616| .000 
Year 8 


Completed any degree by 13.6% |0.360| .000 
Year 9 


Completed an associate’s 3.492 0.314 21.7% 0.8% 20.9% |2.116) .000 
degree or certificate by 
Year 4 


Completed an associate’s 3.123 0.259 23.7% 1.3% 22.3% |1.893] .000 
degree or certificate by 
Year 5 


Completed an associate’s 2.524 0.207 24.5% 2.5% 22.0% /1.530) .000 
degree or certificate by 
Year 6 


Completed an associate’s 2.018 0.171 25.9% 4.4% 21.4% /1.223) .000 
degree or certificate by 
Year 7 


Completed an associate’s 1.605 0.150 27.1% 7.0% 20.2% |0.973) .000 
degree or certificate by 
Year 8 
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Control 
EC Group (clgelt) Effect 
Probability | Probability | Difference | Size 


Effect in 
Logits 


Standard 
Error 


Outcome 


Completed an associate’s 0.837} .000 
degree or certificate by 
Year 9 


Completed a bachelor’s 2.608; .011 


degree by Year 6 


Completed a bachelor’s 1.425; .000 


degree by Year 7 


Completed a bachelor’s 0.461] .000 


degree by Year 8 


Completed a bachelor’s 0.175} .023 


degree by Year 9 


Notes. N = 2,458 (1,044 EC, 1,414 control). The EC group probabilities are unadjusted probabilities; the control 
group probabilities were computed based on the unadjusted EC group probabilities and estimated EC effects. We 
computed the effect sizes for these binary outcomes by dividing the logged odds ratio by 1.65 (i.e., the Cox index), 
as recommended by the WWC. 

Year x, x = 4~10, refers to x years after starting high school. 
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Exhibit F-4. ITT and CATE Estimates of the Overall EC Impacts on Primary Study Outcomes 


(ovata ge)| 


Analysis Standard | EC Group Group Effect 
Outcome Type Error Probability | Probability | Difference | Size 


College Enrollment by Year 10 


Any college ITT 0.464 | 0.121 84.2% 77.0% 7.2% 0.281 | .000 


CATE 0.817 | 0.174 12.9% | 0.495) .000 


2-year college ITT 0.781 | 0.108 19.0% | 0.474] .000 


CATE 1.205 | 0.152 29.0% | 0.730 | .000 


0.8% 0.021) .744 


4-year college ITT 0.035 | 0.106 
CATE 0.135 | 0.144 60.1% 56.8% 3.3% 0.082 | .348 


Degree Completion by Year 10 


Any degree ITT 0.503 | 0.111 45.4% 33.5% 11.9% 0.305 | .000 
CATE 0.748 | 0.144 18.0% 0.453 | .000 
Associate’s degree | ITT 1.197 | 0.132 18.2% | 0.725 | .000 
Cheer ieats caTE | 1.636 | 0.173 24.8% | 0.991 | .000 
Bachelor’s degree ITT 0.259 | 0.124 5.2% 0.157 | .037 
CATE 0.415 | 0.156 33.6% 25.1% 8.6% 0.251 | .008 


Notes. N = 2,458 (1,044 EC, 1,414 control). The EC group probabilities are unadjusted probabilities based on all EC 
students for the intent-to-treat (ITT) findings and based on compliers (i.e., EC students who enrolled in an EC in the 
fall of the first year of high school) for the complier average treatment effect (CATE) findings. The control group 
probabilities were computed based on the unadjusted EC group probabilities and estimated EC effects. 

We computed the effect sizes for these binary outcomes by dividing the logged odds ratio by 1.65 (i.e., the Cox 
index), as recommended by the WWC. 
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Estimates of Differential Impact 


Exhibit F-5. Differential EC Impacts on College Enrollment and Degree Attainment Outcomes by the End of Year 10, 


Differential 
Prior Ach.=1 SD Below Average Prior Ach.=Average Prior Ach.=1 SD Above Average Impact 


by Prior Achievement 


Control Control Control 
EC Group Group EC Group Group EC Group Group Odds| p- 
Outcome by the End of Year 10 | Probability | Probability | Diff. | Probability | Probability | Diff. | Probability | Probability Diff. Ratio | Value 


16.4% |19.8%| 52.4% 18.5% | 33.9% (ea lia 


16.8% 


Completed an associate’s 21.0% 15.7% 16.5% 18.5% | 52.3% 17.4% 34.9% 1.9 
degree or certificate 
Completed a bachelor’s degree 14.0% 10.4% 16.8% 30.9% 26.1% | 4.8% | 0.9 | .742_ 


Notes. Ach. = achievement; Diff. = difference; ELA = English language arts; SD = standard deviation. N = 2,458 (1,044 EC, 1,414 control). The EC and control 
group probabilities for a given level of prior achievement (i.e., 1 SD below the state average, state average, and 1 SD above the state average) are predicted 
probabilities when all control variables other than the prior achievement measure were set to their grand means. The values in the Diff. columns may not 
match the difference between the EC and control group probabilities due to rounding. 


Differential Impact by Prior ELA Achievement 


degree or certificate 

11.7% 
Differential Impact by Prior Mathematics Achievement 
22.2% 
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Exhibit F-6. Differential EC Impacts on College Enrollment and Degree Attainment Outcomes by the End of Year 10, 
by Student Background Characteristics 


Control Control 
EC Group Group EC Group Group Difference 

Outcome by the End of Year 10 | Probability | Probability Diff. Probability | Probability in Impact 
Differential Impact by Gender (X = 1 for female) 
College enrollment 85.8% 80.1% 5.7% |1,260} 82.5% 73.7% 8.8% 1,198} 0.9 -3.1% .656 
2-year college enrollment 69.9% 52.1% 17.8% |1,260) 61.5% 41.7% 19.8% |1,198) 1.0 -2.0% .836 
4-year college enrollment 58.6% 58.9% -0.3% |1,260} 56.4% 54.3% 2.1% -2.4% .656 
Completed any degree 49.4% 38.4% 11.0% |1,260} 41.2% 28.3% 12.8% -1.8% 599 
Completed an associate’s 32.3% 12.2% 20.1% |1,260} 26.1% 10.1% 16.1% 4.0% 774 
degree or certificate 
Completed a bachelor’s degree 33.9% 28.5% 5.4% |1,260} 26.1% 21.2% 4.8% 0.6% 949 
Differential Impact by Race/Ethnicity (X = 1 for non-White) 
College enrollment 84.5% 79.2% 5.3% 954 87.2% 78.5% 8.6% 866 0.8 -3.3% 372 
2-year college enrollment 72.9% 54.5% 18.4% | 954 62.6% 36.8% 25.8% -7.4% 316 
4-year college enrollment 59.8% 56.9% 2.9% 954 59.6% 61.3% -1.7% 4.7% .436 
Completed any degree 44.7% 30.9% 13.8% | 954 53.3% 40.7% 12.6% 866 1.1 1.2% 729 
Completed an associate’s 31.2% 9.4% 21.8% | 954 33.1% 13.0% 20.0% 866 1.3 1.7% 327 
degree or certificate 
Completed a bachelor’s degree 32.1% 24.8% 7.3% 954 33.0% 29.4% 3.6% 866 1.2 3.7% .463 
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Differential Impact 


Control 
LOM CT Co] by e) Group 
Probability | Probability 


Control 
LOM CT Ko] by 0) Group 
Probability | Probability 


Difference 
in Impact 


Outcome by the End of Year 10 


Differential Impact by Low-Income Status (X = 1 for students eligible for free or reduced-price lunch) 


Completed an associate’s 


College enrollment 2.5% .983 
2-year college enrollment 65.0% 44.4% 20.5% /1,313) 66.4% 47.6% 18.8% 1.7% .794 
4-year college enrollment 53.2% 51.4% 1.8% /|1,313) 62.7% 61.9% 0.8% 1.0% .865 
Completed any degree 1.0% .620 


Notes: Diff. = difference. The EC group probabilities within a given student subgroup are unadjusted probabilities; the control group probabilities were 
computed based on the unadjusted EC group probabilities and estimated EC effects within the subgroup. 
The values in the Diff. columns may not match the difference between the EC and control group probabilities due to rounding. Two lotteries were excluded 
from the analyses of differential impact by minority status because zero EC students were nonminority, and thus EC impacts among nonminority students 
could not be estimated within these lotteries. Similarly, one lottery was removed from the analyses of differential impact by low-income status because zero 
students within this lottery were from low-income families. 
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Findings from Mediation Analyses 


Exhibit F-7. Estimates of EC Impacts on High School Experiences Based on the Survey Sample 


Control 
Standard |EC Group} Group Effect 
Outcome Effect Error Mean Mean Difference | Size 


College credit accrual during high | 1.412 | 0.186 69.3% | 35.5% 33.8% |0.856 
school? 


0.357 | 0.074 0.143 | -0.214 0.357 | 0.357 000 | 
0.362 | 0.072 0.136 | -0.225 0.362 | 0.362 000 
0.373 | 0.070 0.159 | -0.213 0.373 | 0.373 000 | 


Note. N = 1,424 (771 treatment, 653 control). The EC group means are unadjusted means; the control group means 
were computed based on the unadjusted EC group means and estimated EC effects. 

@ The EC effect on the binary measure of college credit accrual during high school is on the logit scale; effects on 
the three continuous measures are on a standardized scale with a mean of 0 and a standard deviation of 1. 


Exhibit F-8. Total EC Effects on Primary College Enrollment and Degree Attainment Outcomes 
Based on the Survey Sample 
Control 


Effect in | Standard | EC Group Group 
Outcome Logits Probability | Probability | Difference 


College Enrollment Within 6 Years after Expected High School Graduation 


Any college 0.593 0.118 82.7% 72.5% 10.2% | 0.359 
2-year college 0.894 0.104 62.1% 40.2% 22.0% | 0.542 


4-year college 0.006 0.102 61.1% 60.9% 0.004 


Degree Completion Within 6 Years after Expected High School Graduation 


Bachelor’s degree 


Notes. N = 1,424 (771 EC, 653 control). The EC group probabilities are unadjusted probabilities; the control group 
probabilities were computed based on the unadjusted EC group probabilities and estimated EC effects. 
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Exhibit F-9. Distribution of Direct and Indirect Effects of ECs on Primary College Enrollment and Degree Attainment Outcomes 
Based on the Survey Sample 


Indirect Effects: Percentage of Total Effect Explained by Percentage of 


Percentage of College Credit Total Effect That 
Total Effect That | Accrual During | Instructional | College-Going Is an Indirect 
Outcome Is a Direct Effect High School Rigor Culture Student Supports Effect 


College Enrollment Within 6 Years after Expected High School Graduation 


Any college 71.5% N/A 28.5%** 


2-year college 96.6%*** N/A 


Degree Completion Within 6 Years after Expected High School Graduation 


Any degree 40.9% 50.3%*** 12.0%* 59.1% *** 


Associate’s degree or certificate 78.9%*** 9.8% 21.1%** 


Bachelor’s degree 14.5% 87.3% *** 85.5% *** 


Notes. N = 1,424 (771 EC, 653 control). 

N/A = not applicable; college credit accrual during high school was not included as a mediator for college enrollment outcomes. 

We did not explore direct or indirect effects for the outcome of enrollment in a 4-year college within 6 years after expected high school graduation because the 
total effect on this outcome was not statistically significant. 

* p< .05; ** p< .01; *** p< .001. 
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Appendix G. Sensitivity Analyses 


Comparing Impact Results With and Without the Site With Imputed 


Administrative Data 

For the original Impact Study of Early College High Schools, the research team collected 
administrative data from participating schools, districts, and states. Upon updating data 
agreements at the beginning of this follow-up study in 2016, we were informed that one state 
that provided administrative data no longer allowed researchers to access identifiable student- 
level data. Because student identifiers were necessary to link student background data both to 
survey data and to postsecondary data obtained from NSC, we could no longer use 
administrative data from this state in our analyses, and all student background information was 
imputed for students who participated in the lottery at the one site in this state. 


To assess whether the imputed background information for the students at this site affected 
our overall study findings, we performed a set of sensitivity analyses that estimated EC impacts 
on college enrollment and degree completion outcomes without this site. The results of these 
analyses are presented in Exhibits G-1 and G-2, respectively. Overall, the results from this set of 
sensitivity analyses mirror the results based on the full study sample (see Appendix Exhibits F-1 
and F-3) with one exception. While the impact on bachelor’s degree completion within 6 years 
after expected high school graduation was statistically significant based on the full sample, it 
was no longer significant at the .05 level (but marginally significant at the .10 level) after 
excluding the one site in which all study participants had imputed background data. EC impacts 
on bachelor’s degree completion in previous years remained statistically significant based on 
the reduced sample. 


Exhibit G-1. ITT Estimates of the Overall EC Impacts on College Enrollment Outcomes, 
Excluding One Site With Completely Imputed Student Background Data 


Effect Control 
in Standard| EC Group Group 


Logits tice) g Probability | Probability | Difference 


Effect 
Size 


Outcome 


College enrollment by Year 10 0.314 


2-year college enrollment by 0.499 | .000 
Year 10 
4-year college enrollment by 0.027 | .688 


Year 10 


College enrollment by Year 4 


College enrollment by Year 5 


| 


.000 
.000 


College enrollment by Year 6 
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Effect Control 
Ta] Standard| EC Group Group p- 

Outcome Logits tice) g Probability | Probability | Difference Value 
College enrollment by Year 7 0.125 82.5% .000 
College enrollment by Year 8 0.127 83.3% .000 
College enrollment by Year 9 0.128 83.7% .000 
2-year college enrollment by 0.131 55.2% .000 
Year 4 
2-year college enrollment by 0.108 64.2% .000 
Year 5 
2-year college enrollment by 0.108 68.7% .000 
Year 6 
2-year college enrollment by 0.109 70.5% .000 
Year 7 
2-year college enrollment by 0.109 72.0% .000 
Year 8 
2-year college enrollment by 0.110 72.7% .000 
Year 9 
4-year college enrollment by 0.204 15.3% .000 
Year 4 
4-year college enrollment by 0.115 45.2% .117 
Year 5 
4-year college enrollment by 0.112 50.0% .023 
Year 6 
4-year college enrollment by 0.111 51.5% .179 
Year 7 
4-year college enrollment by 0.110 52.3% .298 
Year 8 
4-year college enrollment by 0.084 | 0.110 53.3% 51.2% 2.1% 0.051 | .444 
Year 9 


Notes. N = 2,260 (920 EC, 1,340 control). The EC group probabilities are unadjusted probabilities; the control group 
probabilities were computed based on the unadjusted EC group probabilities and estimated EC effects. We 
computed the effect sizes for these binary outcomes by dividing the logged odds ratio by 1.65 (i.e., the Cox index), 
as recommended by the WWC. 

Year x, x = 4~10, refers to x years after starting high school. 
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Exhibit G-2. ITT Estimates of the Overall EC Impact on Degree Attainment Outcomes, 
Excluding One Site With Completely Imputed Background Data 
Control 


Effect in Standard | EC Group (clgelt) p- 
Outcome Logits Error Probability | Probability | Difference Value 


Completed any degree by 0.000 
Year 10 


Completed an associate’s 0.000 
degree or certificate by 
Year 10 


Completed a bachelor’s 0.096 
degree by Year 10 


Completed any degree by 0.000 
Year 4 


Completed any degree by 0.000 
Year 5 


Completed any degree by 0.000 
Year 6 


Completed any degree by 0.000 
Year 7 


Completed any degree by 0.000 
Year 8 


Completed any degree by 0.000 
Year 9 


Completed an associate’s 0.000 
degree or certificate by 
Year 4 


Completed an associate’s 0.000 
degree or certificate by 
Year 5 


Completed an associate’s 0.000 
degree or certificate by 
Year 6 


Completed an associate’s 0.000 
degree or certificate by 
Year 7 


Completed an associate’s 0.000 
degree or certificate by 
Year 8 
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Control 
Effect in Standard | EC Group (clgelt) Effect 
Outcome Logits ide) g Probability | Probability | Difference Size 


Completed an associate’s 0.853 
degree or certificate by 
Year 9 


Completed a bachelor’s 21.4% 2.698 
degree by Year 6 
Completed a bachelor’s 53.6% 1.454 
degree by Year 7 
Completed a bachelor’s 9.9% 0.456 
degree by Year 8 
Completed a bachelor’s 5.1% 0.164 
degree by Year 9 


Notes. N = 2,260 (920 EC, 1,340 control). The EC group probabilities are unadjusted probabilities; the control group 
probabilities were computed based on the unadjusted EC group probabilities and estimated EC effects. We 
computed the effect sizes for these binary outcomes by dividing the logged odds ratio by 1.65 (i.e., the Cox index), 
as recommended by the WWC. 

Year x, x = 4~10, refers to x years after starting high school. 


Comparing Results Based on Fixed-Effects and Random-Effects Models 

Exhibit G-3 provides a comparison of results from fixed-effects models and random-effects 
models for the primary college enrollment and degree attainment outcomes. In general, results 
based on the random-effects model align with the results based on the fixed-effects model. 
However, while the EC impact on completion of a bachelor’s degree within 6 years after 
expected high school graduation (i.e., by Year 10) was statistically significant based on the 
fixed-effects model, it was slightly smaller and no longer significant based on the random- 
effects model. 
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Exhibit G-3. ITT Estimates of the Overall EC Impacts on Primary College Enrollment and 
Degree Attainment Outcomes Based on Fixed-Effects and Random-Effects Models 


Control 
Effect | Standard | EC Group Group Effect 
Outcome in Logit Error Probability | Probability | Difference | Size 


College Enrollment by Year 10 


Fixed-effects 0.464 | 0.121 | 84.2% 77.0% 0.281 | .000 | 
Any college 

/Random-effects | effects | 0.464 0.179 84.2% 77.0% 0.281 
aiéar = 0.781 | 0.108 | 65.8% 46.8% 19.0% | 0.474 EJ 
college Random-effects | 0.918 | 0.190 | 65.8% 43.5% 22.4% | 0.556 | .000 | 
Rica Fixed-effects 0.035 | 0.106 | 57.6% 56.7% 0.021 
college Random-effects | 0.044 | 0.136 | 57.6% 56.5% 0.026 

Fixed-effects 0.503 | 0.111 | 45.4% 33.5% 11.9% | 0.305 | .000 | 
Any degree 

Random-effects | 0.449 0.225 45.4% 34.7% 10.7% 0.272 | 046 | 


Associate’s Fixed-effects | effects 1. i stae7 | 0.132 132 29.3% 11.1% 18. | 18.2% | 0.725 | 725 | .000 | 


a aces a effects Ee 968 ee 306 | 29.3% | 13.6% | 415. So es 586 Es 
certificate 


Bachelor’s Fixed-effects | effects | 0.259 | 259 0.124 | 124 30. | 30.1% | 24. | 24.9% | 5.2% | 0.157 | 157 | .037 | 


degree icc ea ce Ere Ee ae 


Notes. N = 2,458 (1,044 EC, 1,414 control). The EC group probabilities are unadjusted probabilities; the control 
group probabilities were computed based on the unadjusted EC group probabilities and estimated EC effects. 
We computed the effect sizes for these binary outcomes by dividing the logged odds ratio by 1.65 (i.e., the Cox 
index), as recommended by the WWC. 


Exhibit G-4 presents the variance of EC impact on each primary study outcome based on the 
random-effects model. It reveals significant variation in EC impact across lotteries for the 
outcomes of completing any type of postsecondary degree and completing an associate’s 
degree or certificate by Year 10 (see Exhibits G-5 and G-6 for a graphic depiction of the 
variation of EC impacts on these two outcomes). For the other primary study outcomes, we did 
not observe significant variation in EC impact across lotteries. 
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Exhibit G-4. Variance Estimates of the Random-Effects Impact Estimates for Primary College 
Enrollment and Degree Completion Outcomes 
Variance of EC 


Effect (logit 
Outcome squared) Standard Error T-Statistic 


College Enrollment by Year 10 


Any college 0.166 0.161 1.033 0.302 
2-year college 0.353 0.200 1.768 0.077 


4-year college 0.104 0.109 0.962 0.336 


Degree Completion by Year 10 


Any degree 0.596 0.302 1.977 0.048 
Associate’s degree or certificate 1.177 0.558 2.107 0.035 
Bachelor’s degree 0.052 0.094 0.547 0.584 


Note. N = 2,458 (1,044 EC, 1,414 control). 


Exhibit G-5. Lottery-Specific EC Impact on Completion of Any Degree by Year 10 Based ona 
Random-Effects Model 


EC impact (odds ratio) 


A-1 B-2 B-3 C-1 C-2 C3 D-2 D-3 E-1 E-3 F-2 F-3 G-3 H-2 I-1 1-2 J-3 


Lottery (site-cohort) 


Note. N = 2,458 (1,044 EC, 1,414 control). 
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Exhibit G-6. Lottery-Specific EC Impact on Completion of an Associate’s Degree or Certificate 
by Year 10 Based on a Random-Effects Model 
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Note. N = 2,458 (1,044 EC, 1,414 control). 
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